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PREFACE 

The  use  of  the  term  "ADP"  or  "system"  in  this  document  is  not  meant 
to  imply  any  particular  functional  category  or  system.  In  particular, 
the  term  is  meant  to  encompass  at  least  the  four  categories  outlined  in 
AFR  800-14:  Category  A--ADP  resources  in  combat  weapon  systems  and 
specially  designed  equipment;  Category  B--ADP  resources  in  other  systems 
developed  under  AFR  800-2;  Category  C--ADP  resources  in  systems 
developed,  acquired,  and  managed  by  AFR  80-2,  AFR  65-2,  AFR  71-11,  and 
AFR  100-2:  and  Category  D--ADP  resources  in  general  purpose,  ADPS 
developed,  acquired,  and  managed  by  the  300-series  regulations  and 
manuals.  Primary  application  of  risk  assessment  tools  and  methodologies 
will  be  to  mission-critical  ADP  systems  covered  by  categories  A  and  B  in 
accordance  with  AFR  800-14. 
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SECTION  I 
INTRODUCTION 


1.1  BACKGROUND. 

The  Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC)  has 

the  responsibility  for  conducting  operational  test  and  evaluation  (OT&E) 
of  assets  entering  the  Air  Force  inventory.  AFOTEC  has  developed  and 

implemented  various  software  OT&E  methodologies.  These  methods  have 

matured  and  have  become  the  Air  Force  standard  for  evaluating  software 
supportabi  1  ity.  Each  of  these  developed  methods  evaluates  specific  char¬ 
acteristics  of  the  supportabi lity  aspects  of  delivered  software  and  soft¬ 
ware  support  resources.  These  stand-alone  evaluations  provide  AFOTEC 
with  information  to  identify  particular  software  supportabi  lity  defi¬ 
ciencies,  but  do  not  identify  overall  risk  associated  with  contractor  or 
military  ownership  and  organic  maintenance  of  contractor-delivered 

software. 

Assessing  the  software  supportabi lity  risk  of  Air  Force  acquired 
systems  is  necessary  to  enable  various  decision  makers  to  properly  plan 
for  system  deployment.  Risk  assessment  (RA)  is  required  throughout  the 
system  acquisition  life  cycle.  The  perspective  of  OT&E  is  focused  upon 
the  overall  system  mission  operation,  including  support.  Methods  are 
needed  to  provide  software  testers  with  areas  whic.i  require  testing 
emphasis,  and  decision  makers  with  an  assessment  of  the  software  support¬ 
abi  1  ity  risk. 

Software  support  for  major  weapon  systems  is  becoming  a  major  system 
cost  factor.  Major  weapon  systems  are  using  more  sophisticated  computer 
systems  and  the  support  costs  required  for  embedded  software  is  projected 
to  increase.  Furthermore,  since  most  enhancements  to  the  system  are 
dependent  on  software  modifications,  the  timeliness  of  such  software 
support  is  critical  to  system  ooerational  availability  and  effectiveness. 
Because  of  tnis  criticality  of  the  software  support  function  to  overall 
system  mission  operational  capability,  it  is  desired  that  top  decision 
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makers  be  aware  of  the  risk  associated  with  the  software  supportab i  1 i ty 
of  a  system  at  the  conclusion  of  OT&E.  In  order  to  determine  this  risk 
during  OT&E,  AFOTEC  needs  to  develop  and  implement  a  risk  assessment 
model  of  software  supportability  with  the  proper  system  mission  perspec¬ 
tive  to  ultimately  assist  the  top  level  decision  maker.  Due  to  the  com¬ 
plexity  of  this  requirement,  it  is  first  necessary  to  determine  the 
feasibility  of  developing  and  implementing  such  a  model. 

AFOTEC  produced  a  concept  proposal  (reference  6.12)  for  computer 
resources  risk  assessment  during  operational  test  and  evaluation.  This 
effort  integrates  an  approach,  appropriate  models,  and  subjective  and 
quantitative  software  operational  and  supportability  measures  into  a 
management-oriented  assessment  of  user  and  supporter  risk.  This  initial 
involvement  with  the  application  of  risk  assessment  to  software  support- 
ability  provided  AFOTEC  with  justification  to  support  a  study  of  the 
feasibility  of  developing  and  implementing  a  risk  assessment  model  for 
software  supportabi  lity  (RAMSS).  The  AFOTEC  Subtask  304  (reference  5.0) 
is  the  statement  of  this  feasibility  study's  objectives  and  required 
reports.  This  report  documents  the  final  part  of  this  study. 

1.2  STUDY  OBJECTIVE. 

The  overall  objective  of  this  task  study,  as  stated  in  Subtask 
Statement  304/00  (reference  6.0),  is  to  perform  a  feasibility  study  to 
determine  the  level  of  effort  and  usefulness  of  developing  and  imple¬ 
menting  a  risk  assessment  model  for  software  supportability  (RAMSS).  The 
first  report  of  the  subtask  (reference  6.31)  documents  the  first  part  of 
the  effort:  to  "review  defense  and  technical  literature  and  current 

research  concerning  methods  of  software  supoortab i  1  i ty  testing  and  risk 
assessment  applicable  to  an  OT&E  environment"  (reference  6.0). 

The  emphasis  for  the  first  part  of  the  task  was  placed  upon: 
a)  Identifying  and  collecting  in'armation 

1)  Literature  search  and  rev  .ew 

2)  Fact-finding  visits/conference 

3)  Contact  with  risk  assessment/software  experts. 
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b)  Assembling  risk  assessment  data  base 

1)  Glossary  of  terms 

2)  Annotated  bibliography 

3)  Key  documents 

4)  Experts/knowledgeable  contacts  list 

5)  Current  research  list. 

The  second  report  of  the  subtask  (reference  6.42)  documents  the 
second  part  of  the  overall  task  study:  "based  on  the  literature  and 
research  review,  analyze  the  feasibility  of  developing  and  implementing  a 
RAMSS  that  could  be  applied  to  the  military  systems  during  AFOTEC- 
conducted  OT&E"  (reference  6.0).  The  emphasis  for  the  second  part  of  the 
task  was  placed  upon: 

a)  Analyzing  current  literature  and  research 

1)  OTIC,  NTIS,  NBS,  RADC,  AFOTEC,  OoO,  periodicals,  etc. 

2)  Potential  RAMSS,  or  parts  of  RAMSS 

3)  Continued  contact  with  risk  assessment/software 

experts. 

b)  Developing  a  potential  framework  for  a  feasible  RAMSS 

1)  RAMSS  framework 

2)  Risk  assessment  methodologies,  techniques,  tools. 

This  report  documents  the  third  and  final  part  of  the  overall  task 

study:  "identify  candidate  measures  of  supportabi lity  risk  for  use  in 

developing  a  feasible  RAMSS.  Provide  an  analysis  of  how  each  measure 
would  support  a  RAMSS  that  could  provide  usable  and  meaningful  results  to 
an  overall  assessment  of  software  supportabi 1  i cy  risk"  (reference  6 .0 ) . 
The  emphasis  for  this  part  of  the  task  was  placed  upon: 

a)  Analyzing  the  candidate  risk  measures  identified  in  parts 
one  and  two  of  this  study 

b)  Selecting  the  appropriate  risk  assessment  methodologies 
which  best  permitted  presentation  of  meaningful  results 

c)  Developing  a  model  framework  which  integrated  the  theory 

of  risk  assessment  with  the  software  evaluation 

methodologies  currently  used  by  AFOTEC 
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d)  Analyzing  the  feasibility  of  developing  and  implementing 
the  proposed  risk  assessment  model. 

1.3  STUDY  APPROACH. 

A  three-step  study  approach  was  adopted  in  Subtask  Statement  304/00. 
The  steps  were: 

a)  Conduct  a  literature  search  and  research  review 

b)  Analyze  the  literature  and  research  information  to  deter¬ 

mine  the  feasibi 1 ity .  of  developing  and  implementing  a 
RAMSS  to  be  applied  to  military  systems  during  AFOTEC- 
conducted  OT&E 

c)  Identify  and  analyze  candidate  measures  of  supportab i 1 i ty 
risk  for  use  in  developing  a  feasible  RAMSS. 

The  first  step  results  are  presented  in  the  report  titled: 
"Software  Supportabi 1  ity  Risk  Assessment  in  OT&E:  Literature  Review, 

Current  Research  Review,  and  Data  Base  Assemblage"  (reference  6.31).  The 
literature  search  and  review  required  identification  of  key  documents 
published  by  governmental  agencies  and  civilian  agencies.  Literature 

searches  of  the  Defense  Technical  Information  Center  (OTIC),  National 
Technical  Information  Service  (NTIS),  and  Rome  Air  Development  Center 
(RAOC)  data  bases  were  conducted.  A  search  and  review  of  National  Bureau 
of  Standards  (N8S)  publications  was  dene.  Key  documents  from  these 

searches  were  identified  and  ordered  for  inclusion  in  the  RA  data  base. 

Several  documents  from  another  AFOTEC  subtask  on  Computer  System  Security 
(reference  6.32)  were  identified.  The  final  report  bibliography  includes 
documents  selected  from  that  list.  Researching  the  available  RA 
technology  also  involved  contact  with  a  number  of  agencies,  and 
identification  of  and  discussions  with  RA  research  and  evaluation 
personnel.  The  basic  form  and  content  of  this  data  base  of  RA 
information  is  described  in  reference  6.31.  <  The  data  base  was  augmented 
and  updated  as  necessary  to  keep  the  data  base  current  throughout  this 
study. 


1-4 


THE  BDM  CORPORATION 


BDM/A-84-565-TR 


The  second  step  results  are  presented  in  the  report  titled: 
"Software  Supportabi 1 ity  Risk  Assessment  in  OT&E:  An  Evaluation  of  Risk 
Methodologies"  (reference  6.42).  Analysis  of  candidate  RAMSS  involved 
analysis  of  literature  and  research  collected  from  step  1  in  the  two 
areas  of  risk  assessment  and  software  supportabi 1 ity.  Very  little 
crossover  between  the  two  areas  was  evident.  Hence,  it  was  important  for 
the  feasibility  requirement  of  this  step  to  analyze  the  elements  of  risk 
assessment,  factors  of  software  supportabi 1 ity,  and  develop  a  framework 
within  which  it  could  be  determined  whether  these  "pieces"  of  a  RAMSS 
could  be  integrated  and  implemented  as  a  RAMSS. 

The  third  step  results  are  presented  in  this  report.  The  analysis 
in  the  areas  of  risk  assessment  and  software  supportabi  lity  performed  in 
step  two  of  this  task  formed  the  basis  for  the  creation  of  the  RAMSS 
model  framework  discussed  in  this  report.  This  framework  incorporates 
the  software  supportabi 1 i ty  evaluation  tools  currently  being  used  by 
AFOTEC,  and  recommends  additional  measurement  data  be  collected  in  the 
area  of  software  support  management.  The  collection  and  representation 
of  measurement  data  are  tied  to  the  theoretical  aspects  of  risk 
assessment. 

1.4  REPORT  ORGANIZATION. 

This  report  is  organized  into  six  sections  plus  a  set  of  appendices 
(acronyms  and  glossary).  Report  sections  satisfy  the  following 
objectives : 

a)  Section  II  contains  an  executive  summary  of  the  analysis 
conducted,  candidate  RAMSSs,  the  RAMSS  evaluation  process, 
development  and  implementation  feasibility  for  the  RAMSS, 
and  conclusions  from  this  study 

b)  Section  III  contains  a  technical  discussion  of  the  RAMSS 
risk  measures.  A  brief  background  presents  the  correla¬ 
tion  of  the  theory  of  risk  assessment  with  the  process  of 
evaluating  software  supportabi 1 ity.  The  RAMSS  risk 
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measures  are  then  discussed  in  more  detail,  particularly 
as  those  measures  apply  to  the  methods  used  for  assessing 
r  i  sk 

c)  Section  IV  contains  a  brief  introduction  to  the  software 

life  cycle  phases  and  discusses  the  assimilation  of  a  risk 
assessment  process,  including  reporting,  with  those  phases 

d)  Section  V  describes  the  feasibility  of  developing  and 

implementing  the  proposed  RAMSS  and  associated  risk 

measures 

e)  Section  VI  lists  the  documents  whose  contents  have  formed 
the  basis  for  this  report 

f)  Appendix  A  lists  acronyms  used  in  this  report 

y)  Appendix  3  is  a  glossary  of  terms  (sources  of  the  terms 

arid  descriptions  are  listed)  used  in  this  report. 
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SECTION  II 
EXECUTIVE  SUMMARY 


2.1  INTRODUCTION. 

This  section  of  the  report  provides  an  overview  of  the  material 
presented  in  sections  III  through  V.  This  overview  summarizes  the  anal¬ 
ysis  conducted,  describes  the  proposed  risk  assessment  model  for  software 
supportabi  1  ity  framework,  identifies  the  selected  measures  of  software 
supportabi 1 ity  risk,  and  discusses  the  development  and  implementation 
feasibil ity. 

2.2  ANALYSIS  CONDUCTED. 

The  material  analyzed  during  this  study  included  documents  obtained 
from  the  Defense  Technical  Information  Center  (DTIC);  the  Rome  Air  Devel¬ 
opment  Center  (RADC);  the  National  Technical  Information  Service  (NTIS); 
Risk  Analysis  (RA)  experts  and  knowledgeable  personnel  contacted  by  tele¬ 
phone,  on  fact-finding  trips  and  at  conferences;  and,  '-eferences  in  key 
documents.  The  first  report  (reference  6.31)  of  this  subtask  contains 
the  list  of  documents  and  sources  which  were  used  as  a  data  base  for  this 
study.  The  major  sources  of  documents,  and  the  document  counts,  are 
given  in  table  2-1.  The  Computer  System  Security  (CSS)  task  listed  below 
includes  data  from  reference  6.33. 

Whereas  a  large  number  of  documents  were  identified  via  the  litera¬ 
ture  search  on  key  words,  it  was  found  that  a  relatively  small  number  of 
documents  actually  applied  to  the  subject  matter  at  hand.  BDM  personnel 
have  obtained  one-half  of  the  total  documents  from  other  (experts,  refer¬ 
ences  in  key  documents,  etc.)  or  in-house  sources.  Of  the  total  of  143 
documents,  approximately  one-fourth  of  them  have  been  identified  as  "key" 
documents,  in  the  sense  that  these  documents  contained  information  which 
was  directly  pertinent  to  the  study  of  risk  assessment  of  software 
supportability  in  the  OT&E  environment.  These  documents  are  listed 
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separately  in  section  VI  of  this  report,  and  form  the  basis  for  much  of 
the  analysis  performed.  Part  of  the  count  difference  between  documents 
ordered  and  received  results  from  giving  a  count  of  one  to  received  docu¬ 
ments  with  more  than  one  volume. 


Table  2-1. 

RA  Data  Base  Summary 


Quantity  of 

Quantity  of 

Quantity  of 

Documents 

Documents 

Documents 

Source  of  Data 

Identified 

Ordered 

Received 

DTIC  (1970-1984) 

450 

5 

3 

NTIS  (1964-1984) 

3000 

53 

38 

RADC 

3200 

21 

9 

CSS  TASK 

16 

16 

15 

AFOTEC 

13 

13 

7 

OTHER/IN  HOUSE 

76 

76 

76 

6755 

184 

148 

2.3  PROPOSED  RAMSS. 

The  conceptual  risk  assessment  model  for  software  supportabi 1 i ty 
(RAMSS)  incorporates  a  theoretical  foundation  for  risk  assessment  with 
software  evaluation  tools  presently  being  used  by  AFOTEC.  The  risk 
assessment  methodologies  chosen  represent  the  authors  opinions  of  the 
most  practical  way  to  assess  risk  under  the  criteria  and  constraints  with 
which  AFOTEC  must  work  and  the  software  evaluation  process  in  general. 
The  AFOTEC  software  evaluation  tools  currently  being  used  are  described 
in  AFOTEC?  800-2,  Volumes  1  through  5.  Only  a  minor  modification  to 
these  software  tools  (mostly  administrative)  would  be  required  to 
integrate  them  with  the  proposed  RAMSS. 

The  risk  assessment  methodology  recommended  involves  the  creation  of 
probability  density  functions  (PDFs)  which  represent  the  probability  that 
a  given  outcome  will  occur.  Important  in  this  concept  is  the  establish¬ 
ment  of  a  baseline  for  the  PDF  by  which  positive  and  negative  outcomes 
can  be  determined  relative  to  that  baseline.  By  proper  representation  of 
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the  PDFs  and  their  associated  baseline,  an  analyst  should  be  able  to 
report  the  risk  magnitude  of  a  given  outcome.  Information  on  the 
severity  of  the  outcome  will  also  be  present. 

The  establishment  of  baselines  is  obtained  by  historical  data  on 
similar  systems  and  is  further  refined  by  user  and  supporter  agreement  on 
the  maintenance  action  requirements  for  supportabi 1 i ty.  These  actions 
may  be  identified  using  the  following  scheme: 


Maintenance  Activities 

Priority  Types 

Complexity  Leve 

Vs 

1.  Correction 

1. 

Normal 

1. 

Low 

2.  Enhancement 

2. 

Urgent 

2. 

Medium 

3.  Conversion 

3. 

Emergency 

3. 

High 

The  combinations 

of  the 

above  actions  yield  various 

possible 

maintenance 

categories,  for 

which 

baseline  " 

values"  must 

be 

defined. 

Data 

are 

collected  for  1) 

requi red 

time  to 

complete  a 

maintenance 

request 

and 

2)  the  number  of 

expected 

changes 

in  a  given 

maintenance 

category 

per 

unit  time. 

Following  the  baseline  establishment,  the  evaluation  of  risk  is 
determined  by  using  closed  form  questionnaires  (current  and  new  AFOTEC 
tools)  to  obtain  the  data  necessary  to  produce  the  probability  density 
functions.  The  evaluation  is  structured  so  the  analyst  can  determine 
detailed  information  about  which  elements  of  supportab  i  1  i  ty  are  driving 
the  risk. 

2.4  RAMSS  EVALUATION  PROCESS. 

To  be  capable  of  maximum  effectiveness,  the  RAMSS  must  be  used 
throughout  the  software  system's  life  cycle.  ’  Such  application  will 
achieve  the  following  benefits: 

a)  Early  planning  and  trade-off  studies  for  software  support 
facility  resource  requirements. 
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b)  Early  visibility  of  user  requirements  for  expected  soft¬ 
ware  support  actions. 

c)  Early  view  of  potential  software  support  management 
problems. 

d)  Capability  to  trace  software  supportabi 1 ity  risk  measures 
throughout  the  life  cycle. 

Proper  application  of  the  RAMSS  evaluation  is  also  required.  This 
application  involves: 

a)  Planning  evaluation 

1)  Establish  baselines 

2)  Tailor  baselines 

3)  Establish  evaluation  structure  for  tools 

b)  Conducting  evaluation 

1)  Calibrate  evaluation  questionnaire  and  evaluators 

2)  Complete  evaluation 

c)  Analyzing  evaluation  results 

1)  Compute  measures  of  risk 

2)  Construct  risk  probability  density  functions 

3)  Compute  support/user  level  of  risk  and  risk  drivers 

4)  P >rf  a  trade-off  analyses 

5)  Determine  evaluation  reliability 

d)  Reporting  results 

1)  To  appropriate  report  level 

2)  Highlight  risk  drivers 

3)  Indicate  alternatives 

4)  Present  consequences 

2.5  DEVELOPMENT  AND  IMPLEMENTATION  FEASIBILITY. 

Throughout  this  study,  there  was  an  effort,  to  identify  an  available 
RAMSS.  No  adequate  models  exist,  hence  an  RAMSS  has  been  proposed  in 
this  report.  The  feasibility  of  developing  this  RAMSS  is  reasonable, 
however  there  are  some  technical  issues  which  need  to  be  resolved  before 
a  full-scale  development  effort  begins.  These  issues  are  described  in 


1 1 -4 


THE  BDM  CORPORATION 


BDM/A-84-565-TR 


more  detail  in  section  V  of  this  report.  None  of  these  issues  is 
considered  unresolvable.  In  summary,  they  may  be  grouped  as  follows: 

a)  Establishing  a  baseline  software  supportabi lity  profile. 
The  establishment  of  a  baseline  profile  is  dependent  upon 
obtaining  historical  data  and  involving  the  user  and 
supporter  inputs.  Depending  upon  the  system,  the  baseline 
may  be  dynamic  (changeable)  up  to  the  final  OT&E  evalua¬ 
tion.  Cooperation  among  user,  supporter,  and  evaluator 
(e.g.,  AFOTEC)  to  establish  this  baseline  could  be  a 
problem  for  the  AFOTEC. 

b)  Development  of  Software  Support  Management  evaluation 
metrics.  The  Software  Support  Management  tool  (recom¬ 
mended  by  this  report)  is  not  currently  implemented  by 
AFOTEC,  and  would  have  to  be  developed.  (The  availability 
of  this  tool  is  not  critical  to  the  full-scale  development 
of  the  RAMSS,  but  it  should  be  developed  and  implemented 
at  a  later  date). 

c)  Upgrade  of  current  software  supportab i 1 i ty  evaluation. 
The  current  tools  used  by  AFOTEC  would  require  modifica¬ 
tion  to  convert  the  evaluation  metrics  to  measures  of 
risk. 

d)  Use  of  operational  effectiveness  measures.  Operational 
effectiveness  measures  such  as  operator  interface,  test 
completeness  and  software  maturity  can  be  developed  for 
inclusion  in  a  model  similar  to  the  RAMSS.  The  risk  base¬ 
line  would  be  different  ( i . e . ,  it  would  be  related  to 
operational  requirements  rather  than  support  require¬ 
ments  ) . 

e)  Interaction  among  evaluation  metrics.  The  proposed  metho¬ 
dology  does  not  consider  interrelationsh ips  between  the 
software  evaluation  factors  in  the  various  tools.  While 
an  independence  does  not  totally  exist,  there  are  methods 
which  should  be  investigated  to  resolve  this  problem. 
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The  resources  necessary  to  develop  and  implement  the  RAMSS  metho¬ 
dology  are  listed  below.  These  resource  estimates  are  preliminary  and 
require  further  refinemen  .< 


Task 

1.  Survey  support  facilities 
for  historical  baseline 
software  support  profile 
data 

2.  Develop  RAMSS  methodology 

3.  Conduct  pi  lot  study 

4.  Refine  methodology 

5.  Develop/acquire/integrate 
requirements  for  automated 
support  tools 


Resource  Requirements 

3  persons,  2  months 

4  persons,  6  months 
3  persons,  1  month 
3  persons,  2  months 
2  persons,  1  month 


2.6  CONCLUSIONS  OF  STUDY. 

a)  No  directly  applicable  RAMSS  exists. 

b)  It  is  feasible  to  develop  an  RAMSS 

c)  The  proposed  RAMSS  uses  the  foundation  of  risk  assessment 
coupled  with  minor  modifications  to  the  tools  for  software 
evaluation  currently  used  by  AFOTEC. 

dl  A  continuation  of  the  development  and  implementation  of 

the  proposed  RAMSS  is  recommended. 
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SECTION  III 

IDENTIFICATION  OF  CANDIDATE  RAMSS  RISK  MEASURES 

3.1  BACKGROUND. 

This  section  discusses  background  information  that  is  essential  to 
understanding  the  risk  assessment  model  for  software  supportabi  1  ity 
(RAMSS)  and  associated  measures  of  risk.  Five  major  topics  are  reviewed 
in  this  section:  1)  the  model  criteria  and  constraints;  2)  the  architec¬ 
tural  principles  of  the  model  framework;  3)  how  risk  theoretical  concepts 
are  integrated  into  the  model;  4)  the  model's  cross-sectional  nature;  and 
5)  the  model's  dynamic  nature.  The  basis  for  the  RAMSS  framework  and 
associated  measures  of  risk  are  described  in  detail  in  the  previous 
report  on  evaluation  of  risk  methodologies  (reference  6.42). 

3.1.1  Criteria  and  Constraints. 

The  report  (reference  6.42)  mentioned  above  presented  some  criteria 
for  an  RAMSS  model  and  associated  AFOTEC  constraints  on  the  use  of  any 
such  model.  Since  these  criteria  and  constraints  greatly  affect  the 
framework  of  the  RAMSS  and  the  associated  measures  of  risk,  they  are 
reiterated  here  (see  table  3-1).  Included  in  the  table  is  a  subjective 
assessment  of  how  well  the  proposed  RAMSS  framework  and  software  support- 
ability  risk  measures  presented  in  section  III  satisfy  the  criteria  and 
constraints. 

3.1.2  Architectural  Principles. 

The  RAMSS  framework  was  developed  with  the  principles  of  risk  theory 
and  the  practices  of  the  AFOTEC  Software  Supportabi 1 ity  Evaluation  in 
mind.  Section  III  describes  the  logical  connections  between  risk  theory, 
AFOTEC  software  supportabi 1 i ty  practices,  and  the  RAMSS  model  framework. 
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Table  3-1. 

Proposed  RAMSS  Capability  to  Satisfy  Criteria  and  Constraints 


RAMSS  CRITERIA 

POOR 

FAIR 

GOOD 

EXCELLENT 

a)  HAS  A  TECHNICAL  DEPTH  AND  RESULT  FORMAT 
APPROPRIATE  TO  ADEQUATELY  ASSISTDECJSION. 

X 

b)  INTEGRATES  AT  LEAST  THE  CURRENT  AFOTEC 
EVALUATION  METHOOOLOGIES. 

X 

c)  HAS  ENOUGH  ACCURACY  AND  REPEATABILITY  TO 
WARRANT  CONFIDENCE  IN  ITS  RESULTS. 

X 

d)  IS  BASED  UPON  A  SOUND  THEORETICAL  SOFTWARE 
AND  RISK  ASSESSMENT  FOUNDATION. 

X 

e)  ALLOWS  FOR  DETERMINATION  OF  WHAT 
ACCEPTABLE  LEVEL  OF  RISK  MEANS  DEPENDING 

UPON  THE  IDENTITY  OF  THE  RISK  AGENT  AND  THE 
SOFTWARE  SUPPORTABIUTY  REQUIREMENTS. 

X 

0  IS  SIMPLE  TO  USE. 

X  i 

AFOTEC  EVALUATION  CONSTRAINTS 

X 

X 

X 

a)  RESOURCE  LIMITATIONS 

1)  PERSONNEL 

2)  TIME 

3)  DATA  COLLECTION  (AVAILABILITY  AND 
ACCURACY) 

b)  VARIABLE  ENVIRONMENT 

1)  COMPUTER 

2)  SOFTWARE 

3)  DEVELOPMENT 

4)  TESTINGTEST  COVERAGE  SCENARIO 

X 

X 

X 

X 

c)  EVALUATION  REPEATABILITY  AND 

UNDERSTAND  ABILITY 

1)  EVALUATOR  EXPERIENCE 

2)  EVALUATION  RELIABILITY 

i!  DEPTH  OF  EVALUATION  MOEs 

1 

1 

X 

X 

X 

d)  INTERNAL  CHARTER 

1)  RESTRICTS  CERTAIN  OVERLAP  AREAS  (R&D) 

2)  EARLY  LIFE  CYCLE  INVOLVEMENT  NOT  WELL 
DEFINED 

i 

X 

X 

76' 
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3.1.3  Integration  of  Risk  Concepts. 

Three  major  concepts  are  inherent  in  the  theoretical  description  of 
risk  (reference  6.42).  First,  risk  involves  outcomes.  Second,  the  out¬ 
comes  are  associated  with  some  probability  of  occurrence.  Third,  the 
notion  of  a  baseline  is  used  to  determine  if  an  outcome  is  an  unwanted, 
negative  consequence.  That  is,  a  baseline  determines  negative  outcomes  J 

from  positive  outcomes. 

First,  let  us  examine  baselines.  Baselines  in  risk  theory  can  be 
logically  connected  to  the  necessary  requirements  for  supporting  a  given 
piece  of  software.  Hence,  if  the  requirements  for  supportabi 1 ity  are  not 
met,  then  a  negative  consequence  will  have  occurred.  For  example,  assume 
that  all  emergency  changes  to  software  are  required  to  be  completed 
within  24  hours.  If  an  emergency  change  actually  takes  43  hours,  then 
this  is  a  negative  outcome.  Thus,  a  baseline,  or  requirement,  must  be 
determined  to  judge  whether  a  given  aspect  of  supportabil  ity  is  negative 
and  undesired.  Baselines  are  a  necessary  condition  for  risk  assessment. 

There  are  several  different  "baselines"  which  are  integrated  into 
the  meaning  of  risk.  There  is  the  risk  baseline,  i.e.,  the  definition  of 
that  profile  of  required  outcomes  against  which  negative  outcomes  (and 
positive  outcomes)  can  be  determined.  There  is  a  software  baseline 
against  which  relative  quality  metrics  can  be  determined.  There  is  the 
OT&E  threshold  baseline  (e.g.,  measures  of  effectiveness)  such  that  any 
outcomes  (e.g.,  evaluation  measures)  below  the  threshold  are  negative. 

It  is  also  possible  to  establish  low  risk  and  high  risk  OT&E  baseline 
metrics  (e.g.,  software  maintainability  score  of  3.3  for  high  risk  and 
5.0  for  low  risk).  In  this  manner  the  outcome  "scale"  can  be  divided 
into  three  regions:  low  risk,  medium  risk,  and  high  risk.  In  the  case 
of  software  supportab i 1 ity  risk  being  developed  in  this  report,  all  these 
"baselines"  will  in  fact  be  integrated  but  on  iy  the  first  (the  risk 
baseline)  will  be  what  is  termed  the  baseline  software  supportabi 1 ity 
(SS)  profile.  This  baseline  is  discussed  in  section  3.2.1. 
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Outcomes  are  what  could  or  does  actually  happen  given  the  character¬ 
istics  of  the  software  maintainability,  the  support  facility  or  the 
support  management.  Outcomes  can  be  actual  or  predicted.  For  the  most 
part  the  outcomes  discussed  in  this  report  refer  to  those  software  main¬ 
tenance  outcomes  predicted  to  occur,  during  the  software  support  phase. 
Hereafter,  mention  of  outcomes  in  this  report  will  thus  mean  predicted 
outcomes,  unless  otherwise  qualified.  Predicted  outcomes  are  a  function 
of  heuristic  estimation,  historical  data,  and  predictor  experience. 
Actual  outcomes  form  the  data  base  of  software  maintenance  activity  at 
any  given  support  facility.  With  each  predictor  outcome,  some 
probability  of  that  outcome's  occurrence  is  attached. 

•  t 

3.1.4  Cross-sectional  Nature  of  RAMSS. 


The  RAMSS  may  be  applied  at  any  point  in  time  during  the  software 
life  cycle.  The  ability  to  create  time-frozen  "snapshots"  of  the  risk 
allows  the  evaluator  to  examine  the  risk  drivers  by  looking  at  the  groups 
of  measures  independently  (under  the  current  methodology).  In  this 
sense,  the  evaluator  is  looking  at  a  "cross-section"  of  the  software 
supportabi 1 ity  evaluation  process.  At  each  snapshot,  baselines  may  be 
reestablished  (perhaps  over  previous  best-guesses)  to  provide  a  more 
accurate  evaluation  The  current  AFOTEC  Software  Supportabi 1 i ty 
Evaluation  scheme  is  the  basis  for  the  RAMSS  model  from  a  cross-sectional 
viewpoint. 

Software  supportabi 1 i ty  can  be  characterized  as  a  hierarchy  of 
evaluation  characteristics.  Software  supportabil  ity  is  first  broken  down 
into  three  main  areas:  the  software  support  facility,  software  product 
maintainability,  and  software  support  management.  Each  of  these  three 
major  areas  has  subcategories.  For  instance,  the  software  support 
facility  includes  personnel,  support  systems,  and  facilities.  (For  a 
complete  description  of  the  AFOTEC  Software  Supportabi lity  structure,  see 
references  6.1  and  6.42.) 

The  hierarchical  structure  of  AFOTEC’s  supportabi lity  scheme 
provides  the  option  of  risk  assessment  being  conducted  at  various  levels 
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of  detail  and  depth.  For  example,  a  relatively  "quick"  risk  analysis  may 
be  required;  in  this  instance,  only  the  higher  level  concepts  in  the 
hierarchy  are  investigated.  If  a  fine-grained,  detailed  risk  analysis  is 
required,  then  the  evaluation  can  be  focused  at  the  lower  level  of  the 
supportabi 1 i ty  hierarchy.  The  hierarchical  nature  also  allows  analysis 
to  be  conducted  at  lower  levels,  yet  be  reported  using  the  more  general 
concepts  found  at  the  higher  levels  in  the  hierarchy. 

3.1.5  Dynamic  Nature  of  RAMSS. 

The  RAMSS  is  dynamic  in  nature.  The  assessment  of  risk  fcr  software 
supportabi 1 i ty  is  done  across  the  software  life  cycle  from  concept 
exploration  through  production  and  deployment.  A  high-level  risk  assess¬ 
ment  of  software  is  required  early  in  the  life  cycle,  preferably  during 
concept  exploration.  Further  risk  analysis  is  required  for  major  changes 
in  system  requirements  which  affect  software  requirements,  and  whenever 
major  decision  points  are  reached  in  the  life  cycle.  This  risk  assess¬ 
ment  may  be  conducted  by  the  using  command,  development  agency,  support 
command,  or  an  independent  agency  (e.g.,  during  IV&V).  The  application 
of  RAMSS  is  based  upon  the  data  available  and  the  desired  level  of  risk 
assessment . 

Further  aspects  of  using  the  RAMSS  during  the  life  cycle  process  are 
discussed  in  section  4.2. 

3.2  RAMSS  RISK  MEASURES. 

The  following  sections  examine  how  a  risk  measure  is  derived  for 
software  supportabi  1  ity.  The  conceptual  framework  for  the  derivation  of 
RAMSS  measures  is  depicted  by  figure  3-1.  (See  reference  6.42  for  a 
complete  discussion  of  the  RAMSS  framework.) 

3.2.1  Baseline  SS  Profile. 

A  baseline  must  be  established  in  order  for  an  assessment  of  risk  to 
be  made.  In  the  software  supportabi 1 ity  context,  the  baseline(s)  can  be 
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equated  with  requirement(s)  for  supportabi 1 i ty  that  have  been  established 
between  user  and  supporter.  (See  section  IV  in  this  report  for  some 
aspects  of  defining  these  requirements).  • 

Baseline  values,  or  requirements,  can  be  established  for  different 
types  of  software  maintenance  actions.  The  types  are  defined  straight 
from  the  accepted  definition  of  software  maintenance.  Recall  that  soft¬ 
ware  maintenance  activities  are  those  for  correction,  enhancement,  and 
conversion  of  the  software.  These  three  actions  can  have  different 
priorities  (depending  on  how  critical  completion  is).  The  priority  types 
conventionally  used  are:  emergency,  urgent,  and  normal .  Further,  asso¬ 
ciated  with  maintenance  action  and  priority  level  is  the  complexity  level 
involved  in  the  maintenance.  Conventionally,  one  of  three  complexity 
levels  are  specified:  J_ow,  medium,  and  high .  A  full  factorial  combina¬ 
tion  of  the  maintenance  action  types,  the  priority  types,  and  the 
complexity  levels,  yields  27  possible  maintenance  categories.  Thus, 
27  requirements,  or  baseline  "values",  must  be  defined;  that  is,  one 
requirement  exists  for  each  maintenance  action-priority-complexity  type. 
A  full  profile  of  requirements  is  simply  the  consideration  of  each 
individual  requirement  all  at  once.  In  other  words,  a  baseline  profile 
is  all  27  requirements  for  maintenance  categories  taken  as  a  whole. 

Several  variables  could  be  used  to  measure  the  baseline,  or  require¬ 
ment,  values.  For  instance,  cost  could  be  used:  a  normal  correction  of 
low  complexity  might  have  a  cost  requirement  (or  baseline)  of 
500  dollars.  Any  expense  greater  than  500  dollars  for  such  a  maintenance 
category  action  would  be  a  negative  outcome. 

The  variables  chosen  here  to  define  the  baseline  value  for  each  of 
the  maintenance  categories  are: 

TC  -  time  (calendar)  to  complete  the  maintenance  request  (input  to 
configured  update). 

NT  -  number  of  change  requests  in  a  given  maintenance  category  per 
unit  time  ( one  year ) . 

That  is,  the  baseline  maintenance  profile  can  be  represented  as  27  pairs 
of  numbers  (TC,  NT).  These  "time"  variables  were  judged  to  be  more 
directly  related  to  maintenance  activity  than  other  choices,  therefore 
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providing  a  more  direct  link  between  software  supportabi 1 ity  evaluation 
metrics  and  measures  of  software  supportabi 1 ity  risk. 

As  an  example,  suppose  a  48-hour  completion  time  requirement  is 
placed  upon  urgent  conversions  of  medium  complexity.  Also,  it  is 
required  that  2  such  urgent  conversions  of  medium  complexity  be  completed 
per  month  (each  within  48  hours).  Then  the  two-dimensional  baseline 
value  is  the  required  time  in  which  each  individual  maintenance  type  must 
be  completed  (48  hours),  and  the  number  of  such  maintenance  types  that 
must  be  completed  per  year  (24).  Figure  3-2  illustrates  the  three- 
dimensional  nature  of  the  baseline  maintenance  profile  and  the  two- 
dimensional  baseline  value  for  each  baseline  maintenance  category. 
Table  3-2  illustrates  a  chart  form  of  the  baseline  maintenance  profile 
data  which  could  serve  as  the  basis  for  a  user  and  supporter  agreement. 

3.2.2  SS  Evaluation. 

The  proposed  software  supportabi ! 1 ty  (SS)  evaluation  follows  the 
current  thinking  and  practices  of  AFOTEC.  That  is,  software  support- 
ability  has  three  major  subcategories:  software  support  facility, 
software  product  maintainability,  and  (as  proposed  in  this  report)  soft¬ 
ware  support  management  (see  figure  3-3).  Each  of  these  three  areas  is 
discussed  in  turn. 

3. 2. 2.1  Software  Product. 

The  expected  outcomes  and  their  probabilities  of  occurrence  for  the 
software  product  are  a  function  (f)  of  the  evaluation  measures,  i.e., 

maintenance  outcome  =  (time  for  maintenance,  number  of  maintenance 

requests) 

=  f  (evaluation  measures) 

Mote  that  the  outcomes  are  defined  using  the  same  variables  as  used  to 
establish  the  baseline.  Thus,  the  outcomes  are  directly  comparable  to 
the  baseline.  The  evaluation  measures  must  be  related  to  risk  levels  to 
complete  the  model.  This  conversion  is  illustrated  in  section  3.2.3. 


1 1 1 -8 


LEGEND. 


THE  BDM  CORPORATION 


BDM/A-84-565-TR 


3dAl 

3DNVN31NIVIAI 


1 1 1-9 


Figure  3-2.  Baseline  ECS  SS  Maintenance  Profile 


THE  BDM  CORPORATION 


BDM/A-84-565-TR 


III-10 


THE  BDM  CORPORATION 


BDM/A-84-565-TR 


I 


INTERFACES 


i - ”» 

I  |  NOT  CURRENTLY  IMPLEMENTED 

I  |  RECOMMENDATION  OF  THIS  REPORT 

I _ 1 


Figure  3-3.  Proposed  Structure  of  Software  Support  Management 
Evaluation 
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The  evaluation  measures  come  directly  from  the  AFOTEC  Software 
Supportabi 1 i ty  Evaluation  scheme.  As  previously  mentioned,  this  scheme 
is  hierarchical  in  nature  so  the  evaluation  measures  are  thus  hier¬ 
archically  structured.  Also,  precise  definitions  exist  for  the  evalua¬ 
tion  measures.  For  example,  at  one  level  in  the  hierarchy 

maintenance  outcome  =  (time  for  maintenance,  number  of  maintenance 

requests) 

=  f  (documentation,  source) 

That  is,  documentation  and  source  code  evaluations  determine  the  measures 
that  determine  the  possible  outcomes  of  maintenance  time  and  maintenance 
requests  along  with  each  outcome's  probability  of  occurrence. 

At  a  lower,  f iner-grained  level  in  the  hierarchy, 

maintenance  outcome  =  (time  for  maintenance,  number  of  maintenance 

requests) 

=  f  (documentation  modularity,  source  code 
modularity,  documentation  descriptiveness, 
source  code  descriptiveness,  documentation 
consistency,  source  code  descriptiveness, 
documentation  consistency,  source  code 
consistency,  documentation  simplicity, 
source  code  simplicity,  documentation 
expandability,  source  code  expandability, 
documentation  instrumentation,  source  code 
instrumentation) 


At  issue  is  whether  the  RAMSS  should  be  conceptually  considered  a 
series  of  single  regression  formats,  i.e., 

maintenance  outcome  =  f  (documentation) 
maintenance  outcome  =  f  (source  code) 


or  whether  the  RAMSS  should  be  conceptually  thought  of  as  in  a  multiple 
regression  format,  i.e. 

maintenance  outcome  =  f  (documentation,  source  code). 
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The  obvious  difference  between  these  two  formats  is  whether  the  risk 
measures  are  considered  separately  (as  in  the  single  regression  format) 
or  considered  together  (as  in  the  multiple  regression  format). 

The  single  regression  format  has  a  problem  because  the  individual 
estimates  of  time  for  maintenance  must  be  combined  in  some  manner  to  come 
up  with  a  composite  estimate  for  product  maintainability.  On  the  other 
hand,  the  multiple  regression  format  is  a  problem  because  the  estimation 
process  becomes  more  difficult  due  to  the  interaction  effects  of  the 
different  dependent  variables  (evaluation  factors).  The  way  in  which  the 
interaction  effects  are  estimated  may  be  different  between  individual 
evaluators.  Thus,  when  some  sort  of  questionnaire  method  is  used,  the 
multiple  regression  format  is  difficult  to  use.  However,  it  is  also  a 
more  realistic  approach. 

3. 2. 2. 2  Software  Support  Facility. 

The  estimation  of  outcomes  and  their  probabilities  for  the  support 
facility  follows  the  same  format  as  is  used  for  software  product  main¬ 
tainability.  Outcomes  and  their  probabilities  are  a  function  of  evalua¬ 
tion  measures.  The  evaluation  measures  come  directly  from  the  AFOTEC 
Software  Supportabi 1  i ty  Evaluation  scheme,  and  are  hierarch ically  struc¬ 
tured  allowing  varying  analysis  and  reporting. 

At  one  level  in  the  hierarchy, 

maintenance  outcome  =  (time  for  maintenance,  number  of  maintenance 

requests) 

=  f  (personnel,  support  systems,  facilities) 

A  further  refinement  in  the  level  of  detail  of  the  evaluation 
measures  can  be  found  in  figure  3-4. 

3. 2. 2. 3  Software  Support  Management. 

The  same  logic  is  used  in  arriving  at  outcomes  and  probabilities  for 
support  management  as  was  used  for  the  other  two  major  supportabi  1  ity 
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categories.  Again,  outcomes  and  probabilities  are  a  function  of  evalua¬ 
tion  measures.  The  evaluation  measures  come  directly  from  the  AFOTEC 
Supportabi  1  ity  Evaluation  scheme.  At  ore  level  in  the  hierarchy,  the 
outcomes  and  probabilities  are  a  function  of  configuration  management  and 
project  management,  i.e., 

maintenance  outcome  =  (time  for  maintenance,  number  of  maintenance 

requests) 

=  f  (configuration  management,  project 
management) 

Configuration  management  is  further  broken  down  into:  identification, 

control,  status  accounting,  and  review/audit.  Project  management  is 
further  broken  down  into:  planning,  project  organization,  design  metho¬ 
dologies,  test  strategies,  and  organization  interfaces  (figure  3-4). 

There  is  currently  no  implementation  by  AFOTEC  of  this  part  of  the 

software  supportabil ity  evaluation.  More  precise  definitions  and  lower 

level  factor  characteristics  would  need  to  be  developed.  The  RAMSS  does 
not  depend  upon  its  use,  nor  does  it  rule  out  inclusion  of  other  possible 
factors.  Some  aspect  of  developing  and  implementing  this  area  of  the 

software  supportabi 1 i ty  evaluation  is  discussed  in  section  5. 

3. 2. 2. 4  SS  Evaluation  Metrics. 

The  vehicle  for  measurement  of  software  supportab i 1 i ty  is  a  closed- 
form  questionnaire.  (See  reference  6.42  for  the  full  details  of  closed- 
form  questionnaires.)  The  point  of  departure  from  previous  AFOTEC 
questionnaires  is  the  emphas i s  that  the  concept  of  the  baseline  is  built 
into  the  questions  themselves.  In  other  words,  the  measurements  of 
software  supportabi 1  ity  are  made  within  the  context  of  the  established 
baseline  requirements  for  supportab i 1 i ty.  In  essence,  the  metrics  are 

gauged  with  respect  to  a  standard:  the  baseline  value. 

Each  evaluation  metric  is  a  measure  of  a  software  supportabi  1  i ty 
characteristic  and  is  determined  by  evaluator  response  to  a  closed-form 
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questionnaire.  The  intent  is  to  phrase  the  question  with  respect  to  the 
capability  to  satisfy  the  given  baseline  requirement.  Thus,  the  evalua¬ 
tion  metric  is  relative  to  the  baseline  value  and  when  properly  converted 
(see  section  3.3)  represents  a  risk  measure  relative  to  this  baseline 
value. 

More  specifically,  an  example  of  an  actual  question  might  be: 

Q.  Based  on  cne  software  support  facility,  the  given  requirements 

can  be  met. 

If  a  particular  requirement  is  that  low  complexity  emergency  enhancements 
be  completed  in  48  hours,  and  10  of  these  maintenance  types  must  be 
completed  per  month,  then  the  question  becomes: 

Q.  Based  on  the  software  support  facility,  low  complexity  emer¬ 

gency  enhancements  can  be  completed  in  48  hours  and  10  of  these 
maintenance  types  can  be  completed  per  month. 

Given  that  there  are  27  possible  maintenance  categories  (action, 
priority,  complexity),  then  the  entire  baseline  profile  might  be  con¬ 
sidered  in  the  question,  e.g., 

Q.  Based  on  the  software  support  facility,  the  baseline  (or 

requ irements)  profile  can  be  met. 

Because  the  evaluation  measures  are  h ierarchical ly  structured,  a 
finer  set  of  metrics  might  be  used  to  pinpoint  where  high  risk  might 
exist  in  the  software  support  facility.  The  question  above  then  becomes 
these  three  questions: 

Q.  Based  on  the  personnel,  the  baseline  (or  requirements )  profile 
can  be  met. 

Q.  Based  on  the  support  systems,  the  baseline  (or  requirements) 

prof i le  can  be  met. 

Q.  Based  on  the  facilities,  the  baseline  (or  requirements)  profile 
can  be  met. 


1 1 1-16 


THE  BDM  CORPORATION 


BDM/A-84-565-TR 


A  given  so.  i^art  proj  ;ct  nay  require  a  very  fine-level  analysis  of 
the  possible  areas  of  high  risk.  The  hierarchical  nature  of  the  evalua¬ 
tion  measures  allows  for  this  more  in-depth  analysis.  For  example,  the 
metric  gauging  personnel  now  becomes: 

Q.  Based  on  the  management,  the  baseline  (or  requirements)  profile 
can  be  met. 

Q.  Based  on  the  technical  personnel,  the  baseline  (or  require¬ 
ments)  profile  can  be  met. 

Q.  Based  on  the  support  personnel,  the  baseline  (or  requirements) 
profile  can  be  met. 

Q.  8ased  on  the  contractor,  the  baseline  (or  requirements)  profile 
can  be  met. 

These  questions,  or  more  accurately,  requirements,  are  presented  to 
one  or  more  evaluators.  Each  evaluator  responds  to  each  question  with  an 
evaluation  score  from  the  scale  below: 


1 

2 

3 

4 

5 

6 

completely 

strongly 

genera  1 ly 

genera  1 ly 

strongly 

completely 

disagree 

disagree 

disagree 

agree 

agree 

agree 

The  scale  or  evaluation  scores  are  assumed  to  be  consistent  across 
the  various  questions.  That  is,  a  score  of  '6'  means  the  same  regardless 
of  which  question  it  is  associated  with.  The  scale  is  also  considered  to 
be  metric  in  that  the  intervals  between  any  two  consecutive  numbers  are 
equal.  This  assumption  of  metric  quality  data  may  be  problematic; 
however,  the  assumption  is  made  as  a  starting  point. 

Ultimately,  the  metrics  must  be  translated  into  some  measure  of 
risk.  This  transf ormation  is  discussed  in  sect. on  3.2.3. 
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3.2.3  Converting  SS  Evaluation  Metrics  to  Risk  Levels. 

The  actual  determination  of  risk  comes  from  the  metrics,  or  evalua¬ 
tion  scores.  First,  in  general  terms,  higher  scores  represent  low  risk 
situations  in  that  the  requirement s)  can  be  met.  Low  scores  provided  by 
the  raters  represent  high  risk  situations.  In  this  instance,  the  base¬ 
line  most  likely  will  not  be  met. 

Because  the  evaluation  scale  is  considered  consistent  across  the 
entire  set  of  evaluation  questions,  areas  of  high  and  low  risk  can  be 
pinpointed.  For  example,  the  raters  may  give  scores  of  *1'  and  '2'  for 
the  questions  dealing  with  the  software  support  facility.  Further, 
scores  of  '5'  and  '6'  may  be  given  for  the  software  product.  In  this 
example,  it  is  obvious  that  the  support  facility  is  the  high  risk  area 
for  supportabi lity. 

The  amount  of  risk  can  be  defined  quantitatively  from  the  evaluation 
score(s).  Specifically, 

R  =  Level  of  risk  =  ^  (evaluation  metric). 


For  a  first  approximation  which  behaves  in  a  manner  as  described  above  we 
define : 


R  = 


1  - 


M-l 

5 


where 


R  =  risk  or  percentage  of  outcomes  that  are  negative 
M  =  evaluation  metric  or  evaluation  score. 

For  instance,  assume  that  an  overall  average  evaluation  score  is 
4.0.  Now,  using  the  formula,  or  transformat  ion ,  given  above,  risk  is 
calculated  as  .40.  (Note  that  when  M  =  3.0,  then  R  =  .60.  The  example 
using  M  =  4.0  produces  R  =  .40  by  coincidence )  That  is,  t^cre  is  a 
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40  percent  chance  that  the  baseline  cannot  be  met.  If  the  requirement  is 
for  maintenance  to  be  completed  in  48  hours,  then  40  percent  of  the  main¬ 
tenance  requests  will  not  meet  the  48-hour  deadline.  Or,  considering  the 
number  of  requests  per  month,  there  is  a  40  percent  chance  that  the 
number  of  requests  per  month  cannot  be  completed.  Table  3-3  illustrates 
the  conversion  of  evaluation  metrics  for  the  indicated  range  of  metrics. 

It  is  convenient  to  think  of  risk  as  a  probability  density  function 
(PDF).  First,  consider  the  case:  time  for  the  completion  of  mainte¬ 
nance.  Given  an  evaluation  score  of  4.0,  we  calculated  that  40  percent 
of  the  outcomes  would  be  negative,  i . e . ,  they  exceeded  the  baseline. 
Figure  3-5  depicts  the  fact  that  40  percent  of  the  possible  outcomes  do 
not  meet  the  requirement  time  of  48  hours.  Conversely,  60  percent  of  the 
outcomes  do  not  exceed  the  baseline. 


Figure  3-5.  Example  Risk  Probability  Density  Function 
(Maintenar.ee  Completion  Time) 

Figure  3-5  gives  us  additional  information  about  the  time  it  takes 
to  perform  maintenance.  More  maintenance  requests  were  made  in  "around 
36"  hours  than  for  any  other  time.  Relatively  few  maintenance  requests 
required  96  hours  to  complete.  Of  all  the  negative  outcomes,  approxi- 
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Table  3-3. 

Proposed  Conversion  of  Metrics  to  Risk  Levels 


SOFTWARE  SUPPORTA8ILITY 
METRIC  RANGE 

RISK  RANGE 

LOW 

RISK 

5.0<M<6.0 

0<R<.2 

MEDIUM 

RISK 

3.3<M<5.0 

,2<R<.54 

HIGH 

RISK 

1.0<M  <  3.3 

54<R<1.0 

GENERAL  CONVERSION  FORMULA: 

M_,  M  =  SOFTWARE  EVALUATION  METRIC 

R  =  1  5  R  =  RISK  ASSOCIATED  WITH  METRIC 
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mately  70  percent  occurred  between  48  and  72  hours.  Thus,  in  this  sense, 
the  magnitudes  of  the  negative  outcomes  are  specified.  Obviously,  the 
magnitudes  depend  on  the  shape  of  the  probability  density  function.  The 
shape  of  the  function  is  not  given  by  a  transformation  of  the  metrics. 
Nevertheless,  some  general  distribution  form  can  be  assumed  or  perhaps 
derived  from  historical  support  data.  Perhaps  a  Poisson  distribution  is 
appropriate,  because  events  occurring  along  a  time  continuum  are  often 
modelled  using  this  functional  form.  All  that  would  need  be  done  is  to 
relate  the  parameters  of  the  Poisson  model  to  an  appropriate  metric. 

Figure  3-6  represents  the  same  risk  ideas,  but  for  the  number  of 
requests  per  month.  Remember  that  negative  outcomes  in  this  case  are 
those  outcomes  which  are  less  than  the  baseline  value.  We  must  be 
careful  to  distinguish  between:  1)  the  case  where  the  number  of  requests 
per  month  established  as  a  requirement  cannot  be  completed,  and  2)  the 
case  where  the  number  of  requests  per  month  did  not  exceed  the  require¬ 
ment.  In  the  first  case,  there  is  a  negative  outcome.  In  the  second 
case,  there  is  no  negative  outcome  (from  the  user  viewpoint  at  least). 
Notice  that  40  percent  of  the  PDF  does  not  meet  the  specified 
requirement. 


40%  of  p  d.f. 


NO.  OF  MAINTENANCE-RE  QUESTS 
COMPLETED  EACH  MONTH  WHEN 
AT  LEAST  10  REQUESTS  ARE  MADE 


Figure  3-6.  Example  Risk  Probability  Density  Function 
(Number  of  Maintenance  Requests) 
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Another  interesting  user/supporter  perspective  can  be  derived  from 
the  probability  density  functions  in  figure  3-5  and  3-6.  The  purpose  of 
the  support  function  is  to  provide  resources  to  satisfy  the  baseline 
maintenance  profile.  When  resources  are  not  sufficient  or  adequate  to 
satisfy  the  baseline,  then  negative  outcomes  to  the  user  (and  supporter) 
are  the  result.  However,  if  the  baseline  requirements  are  overestimated 
(i.e.,  the  number  of  requests  is  actually  less  than  the  baseline  require¬ 
ment,  or  the  stated  maintenance  completion  time  is  less  than  actually 
necessary),  then  this  can  cause  underutilization  of  support  resources. 
This  outcome  can  al so  be  considered  to  be  negative  (at  least  to  the 

support  command).  One  value  of  the  distribution  function,  as  in 

figures  3-5  and  3-6,  is  that  it  provides  a  perspective  on  what  the  range 
of  possible  outcomes  might  be,  so  that  peak  and  minimum  support  resources 
can  be  estimated  and  a  plan  for  distributing  such  resources  can  be 

formulated.  Determining  what  range  is  acceptable  (to  both  user  and 
supporter)  is  a  function  of  the  magnitude  of  the  consequence  rf  a 

negative  outcome  and  the  user/supporter  willingness  to  accept  the  implied 
ri  sk . 

3.2.4  SS  Negative  Outcomes. 

Quite  simply,  negative  outcomes  are  those  outcomes  that  do  not  meet 
the  baseline  requirement.  For  instance,  let  us  assume  that  our  require¬ 
ment  for  low  complexity,  emergency  enhancements  is  that:  (1)  they  be 
completed  in  4.1  hours,  and  (2)  10  such  requests  must  be  c r  ipleted  in  a 
month.  If  a  gi:an  low  complexity  emergency  enhancement  takes  greater 
than  48  hours,  then  a  negative  outcome  has  occurred.  Or,  if  10  such 
maintenance  types  of  this  specification  cannot  be  completed  in  a  given 
month,  then  a  negative  outcome  has  occurred.  As  discussed  in 

section  3.2.3,  there  are  aspects  of  overestimating  baseline  requirements 
which  can  lead  to  negative  outcomes  because  too  many  resources  are  being 
allocated.  Further  consideration  of  these  aspects  is  beyond  the  scope  of 
the  current  report,  but  should  be  considered  during  the  full  development 
of  the  RAMSS  methodology  (see  section  5). 
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3.2.5  SS  Magnitude  of  Consequence 

The  magnitude  of  each  negative  outcome  to  the  user/supporter  risk 
agent  must  also  be  considered.  If  a  requirement  is  48  hours  for  mainte¬ 
nance  completion,  then  negative  outcomes  are  any  maintenance  action 
requiring  more  than  48  hours.  Thus,  negative  outcomes  in  this  example 
can  range  from  just  longer  than  48  hours  to  the  point  where  the  mainten¬ 
ance  action  was  not  even  completed. 

Some  aspects  of  this  magnitude  were  described  as  part  of 
section  3.2.3.  In  general,  the  specific  negative  outcomes  and  their 
consequences  must  be  considered  relative  to  both  user  and  supporter.  We 
have  presented  a  viewpoint  based  primarily  upon  the  user  requirements  (as 
agreed  to  by  the  supporter).  Thus,  any  negative  outcome  which  reflects 
disparity  from  these  requirements  is  both  a  user  and  supporter  risk.  The 
magnitude  of  the  consequence  of  such  risk  to  the  user  or  supporter  may 
well  differ.  In  addition,  some  negative  outcomes  primarily  from  the 
supporter  perspective  have  not  been  explicitly  considered  (see 
sections  3.2.3  and  3.2.4  for  some  details). 
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SECTION  IV 

USING  AN  RAMSS  IN  AN  EVALUATION  PROCESS 

4.1  INTRODUCTION. 

Section  III  describes  elements  of  a  proposed  RAMSS  framework  and 
risk  measures.  To  be  effective,  this  model  must  be  used  as  part  of  an 
overall  software  supportabi 1 i ty  evaluation  process  within  a  well-defined 
risk  management  structure.  Figure  4-1  illustrates  such  a  risk  management 
structure.  The  RAMSS  technical  features  of  SS  T&E  and  SS  Risk  Analysis 
are  described  in  section  III.  This  section  will  briefly  consider  some  of 
the  parts  of  the  evaluation  process  to  illustrate  use  of  the  proposed 
RAMSS.  A  more  complete  description  of  this  process  would  be  part  of  a 
RAMSS  methodology  development  and  implementation. 

4.2  LIFE  CYCLE  PHASES. 

Evaluation  of  software  supportabi 1 i ty  is  a  life  cycle  process. 
There  are  key  events  throughout  a  software  system's  life  cycle  where 
application  of  the  proposed  RAMSS  (or  some  part  of  the  RAMSS)  would  be 
beneficial.  Some  of  these  expected  benefits  throughout  a  typical  life 
cycle  are  summarized  in  table  4-1.  Benefits  which  might  occur  include: 
early  planning  and  trade-off  studies  for  software  support  facility 
resource  requirements;  early  view  of  potential  software  support  manage¬ 
ment  problems;  early  visibility  of  user  requirements  for  expected  soft¬ 
ware  support  actions;  capability  to  trace  software  supportabi 1 i ty  risk 
profile  (i.e.,  measures  of  risk)  throughout  the  life  cycle;  early  view  of 
expected  software  supportabi 1 i ty  risk  drivers;  and,  the  actual  assessment 
of  the  risk  to  user  and  supporter  which  must  be  accepted  before  support 
of  the  software  can  be  assumed. 


IV-1 


THE  BDM  CORPORATION 


BDM/A-84-565-TR 


CsEnsi rivirv  ano  I 

^  COITICAUTV  j 


r 

L, 


A>ii 

CS''«Af'Q« 


1 

J 


r  rci>iOv>c  1 

^  ASfiiMSH'  j 


rtCH  MO  LCGT 


Figure  4-1.  SS  Risk  Management  Model  Framework 
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4.3  EVALUATION  PROCESS  OVERVIEW. 

This  section  is  not  meant  to  replace  development  of  a  RAMSS  method¬ 
ology.  It  will  serve  to  clarify  the  use  of  the  RAMSS  risk  measures  as 

described  in  section  III,  and  as  a  point  of  departure  for  further  method¬ 
ology  development.  There  are  basically  four  parts  to  the  evaluation 
process  as  illustrated  in  figure  4-2. 

4.3.1  Planning  for  the  Evaluation. 

From  the  perspective  of-  an  RAMSS,  the  primary  function  in  the 
planning  phase  is  to  establish  an  appropriate  baseline  SS  profile. 
Because  of  the  level  of  the  evaluation  being  conducted,  it  may  not  be 

necessary  to  consider  a  full  baseline  profile.  A  more  complete 
methodology  should  establish  guidelines  for  collecting  the  baseline  SS 

profile  data  and  then  tailoring  the  profile  to  requirements  appropriate 

for  the  desired  level  of  evaluation.  Tailoring  the  data  might  involve 
"averaging"  the  data  into  a  single  baseline  value  with  a  specified  range 
of  variance.  This  would  greatly  simplify  the  evaluator  effort,  but  might 
add  uncertainty  in  the  accuracy  of  the  evaluation  metrics. 

The  integration  of  the  baseline  SS  profile  as  a  reference  point  for 
each  area  of  evaluation  should  be  established  by  the  more  complete  metho¬ 
dology.  Since  such  a  profile  is  likely  to  have  a  dynamic  form  tailored 
to  each  evaluation  (perhaps  even  parts  of  the  evaluation),  it  is  likely 
that  the  evaluation  questions  should  make  some  generic  reference  to  the 
baseline  SS  profile  as  described  in  the  evaluation  guidelines.  The  level 
of  evaluation  (e.g.,  which  factors  and  the  depth  of  factor  character¬ 
istics  to  be  evaluated)  will  depend  upon  the  phase  of  the  software  life 
cycle  (see  section  4.2)  in  which  the  evaluation  is  being  conducted  as 
well  as  the  criticality  of  software  to  the  mission  function  of  the 
system.  Each  of  the  three  major  areas  of  evaluation  must  be  structured 
so  that  desired  evaluation  metrics  and  associated  measures  of  risk  will 
be  obtained  in  a  form  readily  able  to  be  incorporated  into  SS  evaluation 
reports. 
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Other  planning  considerations  such  as  weighting  factors  and  other 
levels  below  these  factors  must  be  considered  in  light  of  their  direct 
affect  on  risk  assessment.  This  also  can  serve  to  distort  the  identifi¬ 
cation  of  actual  risk  drivers  by  the  bias  inherent  in  a  subjectively 
determined  weight.  If  factors  and  other  level  correlations  are  deter¬ 
mined  through  regression  analysis,  then  there  is  more  basis  for  the 
combination.  Still,  as  long  as  raw  evaluation  metrics  are  available, 
sensitivity  analysis  on  these  combinations  (subjective  weighting  or 
regression)  can  be  done. 

4.3.2  Conducting  the  Evaluation. 

Conducting  the  evaluation  may  occur  over  a  short  or  long  period  of 
time  depending  upon  the  level  of  evaluating  being  conducted.  All  members 
of  the  evaluation  team  (test  planners,  test  managers,  evaluators, 
analysts)  should  be  cognizant  of  the  evaluation  process  and  calibration 
requirements  for  the  evaluation  itself.  It  is  this  calibration  which 
reduces  the  direct  misunderstanding  of  what  is  to  be  evaluated,  reduces 
the  subjectiveness  of  the  evaluation  questions  and  responses,  and 
improves  evaluation  accuracy  and  reliability. 

The  RAMSS  has  very  little  effect  on  complicating  the  already  estab¬ 
lished  AFOTEC  evaluation  itself,  either  technically  or  in  consumption  of 
evaluation  resources  (time  and  people).  The  evaluator  does  not  need  to 
understand  risk  or  what  is  actually  going  to  be  done  with  the  evaluation 
metrics  to  establish  a  level  of  SS  risk.  Thus  the  evaluation  itself  is 
essential^  the  same  as  is  currently  being  used  by  AFOTEC  and  is 
described  in  reference  6.1. 

A. 3. 3  Analyzing  the  Evaluation  Results. 

Most  of  the  application  of  the  RAMSS  will  be  during  the  analysis  of 
evaluation  results.  First,  the  evaluation  metrics  must  be  converted  to 
risk  measures  as  described  in  section  3.2.3.  The  conversion  occurs  at 
each  level  in  the  evaluation  hierarchy  after  the  evaluation  metric  for 
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that  level  has  been  computed  using  the  typical  AFOTEC  weighted  sum  of 
components  technique. 

Since  each  evaluation  question  response  (as  the  average  of  all 
evaluator  responses)  has  a  variance,  there  is  an  associated  variance  in 
the  risk.  This  variance  determines  a  measure  of  SS  risk  and  an  asso¬ 
ciated  risk  range  for  each  level  in  the  hierarchy. 

The  distribution  of  the  risk  around  the  baseline  value  can  be 
estimated  by  using  a  hypothesized  or  actual  historical  SS  profile  proba¬ 
bility  density  function  (see  section  3.2.3),  the  baseline  SS  profile 
requirements ,  and  the  computed  measures  of  SS  risk.  This  helps  provide  a 
perspective  on  the  magnitude  of  the  consequence  of  the  computed  SS  risk. 
It  also  helps  in  the  determination  of  the  effects  of  peak  and  minimum  SS 
resource  loading. 

From  the  computed  measures  of  risk  and  the  associated  risk  proba¬ 
bility  density  function(s),  the  overall  user  risk  and  supporter  risk  is 
computed.  A  complete  RAMSS  methodology  should  provide  guidelines  on  this 
computation. 

The  concepts  presented  in  section  III  focus  upon  the  "sufficiency" 
of  resources,  product  quality,  and  so  forth  to  meet  established  user/ 
supporter  baseline  requirements.  This  establishes  a  common  user  and  sup¬ 
porter  risk.  As  mentioned  briefly  in  section  3.2.3,  there  is  also  the 
added  risk  due  to  the  "necessity"  of  resources,  product  quality  and  so 
forth  to  meet  the  established  user/supporter  baseline  requirements.  In 
this  case,  there  are  too  many  resources  allocated,  or  the  product  quality 
is  too  high  (that  we  should  be  blessed  with  such  a  problem),  to  support 
actual  SS  profile  requirements  because  the  baseline  SS  profile  require¬ 
ments  were  overestimated.  This  implies  a  poor  utilization  of  resources 
has  occurred  with  an  associated  economic  impact. 

The  agent  responsible  for  expending  software  support  funds  (probably 
the  support  command)  would  have  additional  risk  due  to  these  circum¬ 
stances.  A  comparison  of  the  baseline  SS  profile  with  historical  data 
would  help  to  identify  the  more  obvious  overestimations.  Appropriate 
adjustment  of  the  baseline  requirements  would  then  reduce  the  risk  due  to 
this  situation.  However,  the  predominant  focus  is  on  "sufficiency",  and 
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unless  the  evaluation  questionnaire  structure  is  modified  to  consider 
both  sufficiency  and  necessity,  there  will  be  some  supporter  risk  not 
directly  considered.  Since  the  modification  is  perceived  to  be  very  com¬ 
plex,  and  the  associated  analysis  of  evaluation  results  even  more 

complex,  this  approach  has  not  been  incorporated  into  the  RAMSS. 

From  the  computed  measures  of  risk  and  the  estimated  risk  probabil¬ 
ity  density  functions,  it  is  possible  to  perform  simple  trade-off  studies 
and  sensitivity  analysis.  For  example,  one  can  identify  directly  the 

reduction  in  risk  due  to  an  improvement  in  an  evaluation  component.  It 
may  be  possible  to  significantly  reduce  risk  by  appropriate  upgrade  of  a 
few  software  support  risk  drivers.  The  possible  tradeoffs  to  reduce  SS 
risk  is  also  easy  to  explain  and  is  ideal  for  including  in  reports  for 
top  level  decision  makers. 

By  having  a  reasonably  concise  and  simple  representation  of  SS  risk, 
the  distribution  of  risk  about  the  agreed-upon  baseline  SS  profile,  and 
alternative  choices  to  reduce  risk,  it  is  a  straightforward  task  to 

present  this  information  to  the  user  and  supporter  for  acceptance.  It 
may  be  that  the  level  of  risk  of  not  being  able  to  adequately  support 
emergency  requests  in  a  unit  time  period  is  acceptable  to  the  supporter, 
but  not  to  the  user.  In  this  case,  either  the  baseline  SS  profile  must 
be  changed  (unlikely  in  this  case),  or  the  measures  of  risk  must  be 

reduced  (i.e.,  the  characteristics  being  evaluated  must  be  improved). 
The  alternative  approaches  to  reducing  SS  risk  can  thus  be  easily  iden¬ 
tified,  supported  by  evaluation  data  and  tne  RAMSS,  and  implemented  with 
reasonable  assurance  of  an  effective  outcome.  Furthermore,  this  process 
can  be  easily  explained  to  the  appropriate  1evel  of  decision  makers. 

4.3.4  Reporting  the  Results  of  an  Evaluation. 

Perhaps  the  most  important  aspect  of  the  RAMSS  is  the  direct  rela¬ 
tionship  of  the  evaluation  metrics  to  the  SS  measures  of  risk  and  alter¬ 
native  choices  to  reduce  SS  risk  which  need  to  be  reported.  The  con¬ 
sequences  (in  particular,  magnitude  of  particular  negative  outcomes,  and 
support  resource  commitments  to  reduce  the  number  of  negative  outcomes) 
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of  the  measures  of  risk  are  not  quite  as  directly  understood  or  report- 
able.  A  more  complete  RAMSS  methodology  would  provide  guidance  as  to  the 
estimation  of  these  consequences. 

The  various  levels  of  reporting  and  report  constraints  for  AFOTEC  is 
described  in  reference  6.40.  The  report  types  are  discussed  in  relation¬ 
ship  to  the  RAMSS  in  reference  6.42.  The  reporting  of  risk  information 
derived  from  the  integration  of  the  RAMSS  with  a  software  supportabi 1 i ty 
evaluation  is  illustrated  in  table  4-2. 
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SECTION  V 

FEASIBILITY  OF  DEVELOPING  AND  IMPLEMENTING  PROPOSED 
RAMSS  AND  ASSOCIATED  MEASURES  OF  RISK 


5.1  INTRODUCTION. 

The  report  on  Evaluation  of  Risk  Methodologies  (reference  6.42) 
included  a  preliminary  analysis  of  the  feasibility  of  developing  and 
implementing  a  RAMSS.  This  section  extends  that  effort  to  include  a  more 
refined  view  of  the  proposed  RAMSS  and  the  associated  measures  of  risk 
presented  in  this  report.  In  addition,  there  are  now  some  reasonable 
technical  issues  which  need  to  be  discussed  as  explanations  of,  and  in 
some  cases  caveats  to,  the  feasibility  of  accomplishing  such  a  develop¬ 
ment.  These  technical  issues  are  presented  in  section  5.2  as  a  summary 
of  this  feasibility  analysis.  The  development  and  implementation  of  the 
proposed  RAMSS  with  associated  measures  of  risk  is  described  in  terms  of 
the  development  tasks,  and  serves  as  a  guideline  for  the  level  of  effort 
required  to  accomplish  the  tasks. 

5.2  FEASIBILITY  ANALYSIS. 

5.2.1  Establishing  a  Baseline  SS  Profile. 

There  will  always  be  technical  problems  in  establishing  definitive 
requirements  such  as  the  Baseline  SS  Profile.  The  importance  of 
obtaining  this  data  is  critical  to  the  use  of  the  proposed  RAMSS. 

It  has  been  assumed  that  the  best  approach  is  for  the  user  to 
"quickly"  estimate  the  full  profile  during  the  software  concept  explora¬ 
tion  life  cycle  phase.  This  estimate  could  be  accomplished  using 
historical  profile  data  (if  such  data  is  available),  a  user's  personal 
experience,  and/or  support  command  suggestions.  Other  sources  of  data 
for  the  users  are  possible.  It  may  be  necessary  for  the  user  and 
supporter  to  combine  knowledge  and  produce  an  initial  cut  at  the  data. 

It  is  also  possible  that  a  heuristic  estimate  by  the  evaluation 
agency  (e.g.,  AFOTEC)  may  be  necessary  using  the  same  sources  as  above, 
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but  with  the  added  burden  of  obtaining  some  level  of  agreement  on  the 
data  values  with  both  the  user  and  supporter. 

In  any  case,  it  should  be  stressed  that  the  baseline  SS  profile  is 
expected  to  be  somewhat  dynamic  up  to  the  final  OT&E  evaluation.  Changes 
can  be  expected  throughout  the  life  cycle  based  upon  new  evaluation 
information  and  (perhaps)  technological  advances.  It  should  therefore  be 
clear  that  temporary  acceptance  of  such  a  profile  by  the  user  and/or 
supporter  does  not  commit  the  user  and/or  supporter  to  that  profile 
forever.  Hopefully,  this  baseline  profile  will  evolve  to  be  the  "best 
guess  estimate"  of  the  actual  SS  profile  data  to  be  collected  during  the 
operation  and  maintenance  phase.  These  caveats  should  make  it  easier  to 
obtain  user  and  supporter  agreement  so  the  further  steps  in  the  risk 
assessment  process  can  be  taken. 

One  aid  to  understanding  this  baseline  profile  would  be  an  empirical 
data  base  gathered  from  actual  support  centers.  This  data  collection 
task  is  suggested  as  part  of  the  RAMSS  methodology  development. 

Although  there  are  some  concerns  about  the  technical  (as  well  as 
logistical  and  political)  feasibility  of  obtaining  an  accurate  baseline 
SS  profile,  it  is  not  resource  bound.  It  is  simply  a  matter  of  making 
reasonable  estimates.  It  is  expected  that  this  will  be  feasible  within 
the  refinements  which  will  result  from  the  complete  RAMSS  methodology 
development. 

5.2.2  Development  of  Software  Support  Management  Evaluation  Metrics 

This  area  of  evaluation  is  not  currently  implemented  by  AFOTEC.  It 
is  a  recommendation  of  this  report  that  a  factor  hierarchy  similar  to  the 
other  current  AFOTEC  software  supportabi  1  i ty  evaluations  be  developed  for 
software  support  management.  It  is  not  critical  to  the  operation  of  the 
proposed  RAMSS,  but  such  an  implementation  should  provide  invaluable  data 
concerning  the  software  support  management  concerns  across  the  life  cycle 
process. 

The  feasibility  of  accomplishing  the  development  of  these  evaluation 
metrics  is  very  good.  There  are  no  known  technical  restrictions  and  the 
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emphasis  in  this  area  is  justified  by  the  recent  software  maintenance 
literature  (see  reference  6.31  and  6.42).  It  should  be  emphasized  that 
factors  and  characteristics  in  this  area  would  cover  software  management 
procedures,  practices,  standards,  and  so  forth  in  both  development  and 
maintenance  phases  of  the  software.  Frequently  the  management  function 
in  both  phases  (including  the  transition)  is  very  important  to  the 
overall  software  supportabi 1 i ty  cohesion.  The  major  factors  of  project 
management  and  configuration  management  (see  section  3. 2. 2. 3),  besides 
capturing  the  essence  of  software  support  management,  illustrate  this 
requirement  for  cohesion.  If  implemented,  there  would  be  additional 
requirements  for  evaluation  resources.  With  some  care,  this  requirement 
should  be  within  current  AFOTEC  constraints. 

5.2.3  Upgrade  of  Current  Software  Supportabi 1 i ty  Evaluation. 

Because  of  the  necessity  to  evaluate  relative  to  a  baseline  require¬ 
ment  (i.e.,  the  baseline  SS  profile)  in  order  to  convert  the  evaluation 
metrics  to  measures  of  risk,  the  current  evaluation  questionnaires  and 
evaluation  procedures  would  have  to  accommodate  this  change.  Since  this 
change  can  be  taken  care  of  in  an  administrative  fashion  (e.g.,  guide¬ 
lines  to  the  evaluators),  at  least  for  the  initial  RAMSS  methodology 
development,  the  impact  upon  the  current  mode  of  operation  would  be 
minimal.  Integration  of  the  refined  methodology  into  the  current  evalua¬ 
tion  process  would  be  straightforward  and  might  be  combined  witn  a  more 
general  upgrade  of  the  evaluation  questionnaires  and  procedures,  or 
separately  done. 

5.2.4  Use  of  Operational  Effectiveness  Measures. 

The  operational  effectiveness  measures,  such  as  the  operator  inter¬ 
face  evaluation  measures,  test  completeness  measures,  and  software 
maturity  measures,  have  not  been  discussed  in  any  detail.  It  is  possible 
to  use  these  measures  as  part  of  the  input  to  the  proposed  RAMSS. 
Similar  considerations  for  other  AFOTEC  evaluations  apply,  but  the 
proposed  RAMSS  must  be  applied  indirectly. 
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It  should  be  emphasized  that  these  effectiveness  measures  do  not 
indicate  either  user  or  suDporter  risk  relative  to  the  specified  baseline 
SS  profile.  These  metrics  indicate  something  about  the  current  opera¬ 
tional  status  of  the  software.  If  the  evaluation  metrics  are  poor,  then 
one  might  suspect  there  will  be  more  software  supportabi 1 i ty  risk  rela¬ 
tive  to  the  baseline  SS  profile  (which  is  presumably  based  upon  software 
meeting  specification  requirements  and  having  a  good  operational  status). 
This  in  turn  would  cause  the  user  to  have  more  risk  relative  to  the 
agreed  upon  baseline  SS  profile.  Note,  however,  that  the  operational 
effectiveness  evaluation  metrics  are  not  relative  to  the  baseline  SS 
profile,  but  are  relative  to  some  other  baseline  operational 
requi rements . 

The  integration  of  operational  effectiveness  measures  might  be 
considered  during  the  development  of  the  RAMSS  methodology.  At  this  time 
this  integration  does  not  appear  to  be  straightforward .  The  issue  of 

having  multiple  baselines  against  which  an  evaluation  is  conducted  can  be 
confusing  and  derived  metrics  are  not  directly  comparable. 

5.2.5  Interrelationships  Among  Evaluation  Metrics. 

In  section  III  there  was  some  discussion  about  the  assumption  of 
independence  of  factors,  with  the  caveat  that  interrelationships  do,  in 

fact,  exist.  The  usual  technique  to  determine  interrelationships  is  to 

use  parametric  methods  such  as  factor  analysis  and  regression  analysis. 

The  current  AFOTEC  technique  is  to  simply  take  a  weighted  sum  of 

components  to  determine  the  evaluation  metric  at  each  level.  Due  to  the 
complexity  of  parametric  analysis  without  known  validated  data,  it  is 
recommended  that  AFOTEC  retain  the  weighted  sum  concept  as  part  of  the 
RAMSS  methodology.  However,  during  the  proposed  support  facility  data 
collection  effort  and  pilot  study  to  establish  a  historical  baseline  S3 
profile,  it  is  recommended  that  parametric  analysis  be  used  to  study 
possible  interrelationships.  It  is  not  anticipated  that  such  analysis 
will  alter  the  RAMSS  methodology  development. 
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5.3  DEVELOPMENT  LEVEL  OF  EFFORT. 

The  recommended  tasks  and  associated  level  of  effort  to  develop  a 
RAMSS  methodology  are  illustrated  in  table  5-1.  The  development  tasks 
include:  collection  of  data  from  current  software  support  facilities  to 
establish  a  historical  baseline  SS  profile;  completion  of  the  theoretical 
foundation  of  the  RAMSS;  upgrade  of  the  software  supportabi  1  i ty  evalua¬ 
tion  methodology  to  incorporate  the  RAMSS;  a  pilot  study  using  a  pre¬ 
liminary  guidebook  for  assessing  software  supportabi  1  i ty  risk  using  the 
RAMSS;  and  refinement  of  the  RAMSS  methodology  and  guidebook  based  upon 
the  pilot  study  "lessons  learned." 

The  task  to  integrate  requirements  for  automated  support  tools  is 
primarily  for  risk  assessment  analysis  and  the  supportabi 1  i ty  evaluation 
tools  which  already  exist  and  are  used,  but  which  are  not  integrated  into 
a  cohesive  system  of  tool  support.  These  tools  are  used  to  provide  the 
necessary  statistical  analysis,  risk  computation,  and  bookkeeping  func¬ 
tions  which  will  be  part  of  the  RAMSS  methodology. 

It  should  be  emphasized  that  the  projected  level  of  effort  (resource 
requirements)  is  a  preliminary  estimate  at  this  time. 
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ADP 

AFOTEC 

CRISP 

CSS 

DoD 

DTIC 

DT&E 

ECS 

IV&V 

NBS 

NTIS 

OT&E 

O/S  CMP 

PDF 

RA 

RADC 

RAMSS 

SS 

TEMP 

T&E 


APPENDIX  A 
ACRONYMS 

Automatic  Data  Processing 

Air  Force  Operational  Test  and  Evaluation  Center 

Computer  Resources  Integrated’  Support  Plan 

Computer  System  Security 

Department  of  Defense 

Defense  Technical  Information  Center 

Developmental  Test  and  Evaluation 

Embedded  Computer  System 

Independent  Verification  and  Validation 

National  Bureau  of  Standards 

National  Technical  Information  Service 

Operational  Test  and  Evaluation 

Operational/Support  Configuration  Management  Plan 

Probability  Density  Function 

Risk  Assessment 

Rome  Air  Development  Center 

Risk  Assessment  Model  for  Software  Supportabi 1 i ty 

Software  Supportabi 1 i ty 

Test  and  Evaluation  Master  Plan 

Test  and  Evaluation 
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APPENDIX  B 
GLOSSARY  OF  TERMS 


B.l  INTRODUCTION. 

The  glossary  of  terms  for  the  Analysis  of  Software  Supportabi 1 ity 
Risk  Assessment  Models  varied  as  the  project  progressed.  Refer  to  BDM/A- 
84-322-TR,  (Final)  dated  September  23,  1984,  for  the  complete  glossary  of 
terms. 

Some  terms  have  more  than  one  description;  when  this  is  the  case, 
the  descriptions  either: 

a)  Are  significantly  different  between  sources  (though  the 
effective  meaning  may  be  not  much  different). 

b)  Are  used  differently  (different  users  or  technical  langu¬ 
age). 

c)  May  be  found  within  the  context  of  a  different  source. 

d)  Have  real  differences  in  meaning. 

Both  DoD  and  non-DoD  (e.g.,  FIPS  PUBs,  NBS  Special  Publications)  sources 
are  used.  The  non-DoD  sources  and  terms  are  not  mandated  for  our  use, 
but  are  rather  included  for  breadth  of  understanding,  for  those  relevant 
terms  commonly  used  within  the  non-DoD  governmental  and/or  private 
sectors. 

The  source  of  each  description  is  indicated  by  a  symbol  in  paren¬ 
theses  before  chat  source's  term  description: 

TERM1 

(SYMB0L1  ,) 

Description1  .... 

(SYMBOL,  2f 
Description^  . . 

(SYMBOL^) 

Description^  n> . . 
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term2 


The  symbols  used  and  corresponding  sources  are: 


(AF0TECP1)  AFOTECP  800-2,  Volume  1,  10  Nov  82,  "Software  Test 

Manager's  Guide." 

(AF0TECP3)  AFOTECP  800-2,  Volume  III,  1  Jan  84,  "Software  Maintain¬ 

ability  Evaluator's  Guide." 

(AFR800-14)  Air  Force  Regulation  800-14,  Volume  I,  "Management  of  Com¬ 
puter  Resources  in  Systems,"  12  Sep  75. 

(AFR300-15)  Air  Force  Regulation  300-15,  "Automated  Oata  System 

Project  Management,"  Jan  78. 

(AF0TECP5)  AFOTECP  800-2,  Volume  5,  25  Jul  83,  "Software  Support 

Facility  Evaluation--User ' s  Guide." 

(ROWE)  Rowe,  William,  An  Anatomy  of  Risk,  John  Wiley,  1977. 

(LATHROP)  Lathrop,  Frank,  "Alternative  Methods  for  Risk  Analysis:  A 

Feasibility  Study,"  Air  Force  Computer  Security  Program 
Office,  1  Sep  81. 

(AFR205X)  Air  Force  Regulation  205-16,  "Automatic  Data  Processing 

(AOP)  Security  Policy,  Procedures  and  ResDonsibi 1 ities, 
1  Aug  84. 

(CURRENT)  Current  document  definition. 
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B.2  GLOSSARY  OF  TERMS  FOR  THE  ANALYSIS  FOR  DETERMINING  FEASIBILITY 
OF  DEVELOPING  AND  IMPLEMENTING  A  RISK  ASSESSMENT  MODEL  FOR 
SOFTWARE  SUPPORTABILITY . 

Accuracy 

(ROWE) 

The  quality  of  being  free  from  error.  The  degree  of  accuracy  is  a 
measure  of  the  uncertainty  in  identifying  the  true  measure  of  a 
quantity  at  the  level  of  precision  of  the  scale  used  for  the  quan¬ 
tity. 

Algorithm 

(AF0TECP3) 

A  prescribed  set  of  well-defined  rules  or  processes  for  the  solution 

of  a  problem  in  a  finite  number  of  steps. 

* 

Allocated  Baseline 
(AFR300- 15) 

The  initial  approved  allocated  conf iguration  identification  estab¬ 
lished  at  end  of  the  definition  phase. 

A1 ternati ve 

(ROWE) 

One  member  of  a  set  of  options  associated  with  a  decision,  tne 
decision  piling  limited  to  a  choice  of  one  and  only  one. 

Application  Functions 

(AF0TECP3) 

Any  functions  wnich  provide  specific  operational  (mission)  computa¬ 
tions. 

Application  Software 
(AF0TECP5) 

The  software  written  oy  software  supocrc  oersonnel,  or  ourchased 
from  a  contractor,  used  directly  ^n  supporting  ECSs.  It  is  normally 
used  for  simulation,  testing,  and  ECS  code  develocment. 

Application  Software  (funct;cnal) 

(AFR205X) 

Those  routines  and  programs  designed  by  or  for  automatic  data 
processing  system  users  ana  customers  to  complete  specific,  mission- 
oriented  task,  jobs,  or  functions,  using  available  automated  data 
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processing  equipment  and  basic  software.  Aoolication  Software  may 
be  either  general  purpose  packages,  such  as  demand  deposit  account¬ 
ing,  payroll,  machine  tool  control,  etc.,  or  specific  application 
programs  tailored  to  complete  a  single  or  limited  numoer  of  user 
functions  (for  example,  base  level  personnel,  depot  maintenance, 
aircraft,  missile  or  satellite  tracking,  command  and  control,  etc.). 
Except  for  general  purpose  packages  that  are  acquired  directly  from 
software  vendors  or  from  the  original  equipment  manufacturers,  this 
type  of  software  is  generally  developed  by  the  user,  either  with  in- 
house  resources  or  through  contract  services. 

Approval  to  Operate 

(AFR205X) 

Represents  concurrence  by  the  designated  approving  authority  (DAA) 
that  a  satisfactory  level  of  security  (that  is,  minimum  requirements 
are  met  and  an  acceptable  level  of  risk  exists)  has  been  provided, 
and  authorizes  the  operation  of  an  automated  data  processing  system 
(AOPS)  or  network  at  an  automatic  data  processing  facility  (AOPF). 
Approval  results  from  an  analysis  of  the  AOPF,  ADPS,  and  automatic 
data  system  (ADS)  certifications  and  tne  operational  environment  of 
the  automatic  data  processing  (AOP)  entity  by  the  OAA. 

Attributes 

(AF0TECP3) 

Type,  units,  range,  description,  etc.,  as  appropriate. 

Automated  Oecisionmaking  System 
(AFR205X) 

Those  computer  applications  which  issue  checks,  requisition  sup¬ 
plies,  or  perform  similar  functions  based  on  programmed  criteria, 
with  little  human  intervention. 

Automated  Software  development  Tool 

(AF0TECP5) 

A  component  of  System  Software  that  assists  in  the  design,  imple¬ 
mentation,  documentation,  and  verification  of  ECS  software. 

Automatic  Data  Processing  Facility  (ADPF) 

(AFR205X) 

The  pnysical  resources,  ‘'rc'uding  structures  or*  carts  O”'  structures, 
wnicn  house  and  support  data  processing  caoaoi 1 i t i es .  -or  eacn  com¬ 
puter  facility  designated  as  a  data  orccessing  installation  (DPI, 
reference  AFR  300-6),  the  ACPC  is  tne  DPI.  For  small  computers, 
stand-alone  systems,  and  word  processino  ecuipment,  the  ADPF  is  tne 
physical  area  in  wnicn  the  computer  is  used. 
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Automatic  Data  Processing  Resources 
(AFR205X) 

the  totality  of  automatic  data  processing  equipment,  software,  data, 
computer  time,  computer  programs,  automatic  data  processing  (ADR) 
contractual  services,  ADP  personnel,  and  supplies. 

Avai labi 1 ity 

(AFR800- 14) 

A  measure  of  the  degree  to  which  an  item  is  in  the  operable  and 
commitable  state  at  the  start  of  the  mission,  when  the  mission  is 
called  for  at  an  unknown  (random)  point  in  time.  (MIL- STD- 7 21) 

(AF0TECP5) 

The  probability  that  a  system  is  operating  satisfactorily  at  any 
point  in  time  when  used  under  stated  conditions. 

Basel i ne 

(AFR300-15) 

A  configuration  identification  document  or  set  of  such  documents 
formally  designated  and  fixed  at  a  specific  time  during  a  CPCI's 
life  cycle.  Baselines,  plus  approved  changes  to  those  baselines 
constitute  the  current  configuration  identification. 

(ROWE) 

A  known  reference  used  as  a  guide  for  further  development  activi¬ 
ties. 

Baseline  Profi le 
(CURRENT) 

The  set  of  27  pairs  of  numbers  (or  any  subset)  determined  by 
specifying  the  (time  to  complete  request,  number  of  requests  per 
unit  time)  pair  for  each  maintenance  request  category. 

Bayesian  Statistics 

(ROWE) 

"Bayes  Rule"  (Thomas  Bayes,  a  nineteenth  century  English  mathematic¬ 
ian  and  clergyman)  states  that  the  probability  that  both  of  two 
events  will  occur  is  the  probability  of  the  first  multiplied  by  the 
probability  that  if  the  first  has  occurred,  the  second  will  also 
occur.  Bayesian  statistics  is  a  way  of  making  quantity  of  informa¬ 
tion  substitute  for  quality  of  information.  There  are  two  kinds  of 
probability:  the  classical  type  derived  from  empirical  information, 
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and  subjective  probability.  Bayesian  statistics  is  based  on  these 
"subjective  probabilities."  It  involves  the  joint  probability  of  A 
and  B.  The  probability  of  the  second  event  occurring  if  the  first 
has  occurred  is  called  the  conditional  probability  of  the  second, 
given  the  first.  Stated  another  way,  the  probability  of  any  event 
P(A)  is  always  positive  but  never  greater  than  1.  Symbolically,  0  <_ 
P(A)  <_  1.  If  P(A)  =  0,  the  occurrence  of  the  event  B  is  considered 
impossible.  If  P(A)  =  1,  the  occurrence  of  the  event  B  is  con¬ 
sidered  to  occur  with  P(B). 

Benefit 

(ROWE) 

a)  An  axiological  concept  representing  anything  received  that  causes 
a  net  improvement  to  accrue  to  the  recipient. 

b)  A  result  of  a  specific  action  that  constitutes  an  increase  in  the 
production  possibilities  or  welfare  level  of  society. 

Benefit-Cost  Ratio 

(ROWE) 

The  ratio  of  total  social  benefit  to  total  social  costs  related  to  a 
specific  activity. 

Capability 

(ROWE) 

A  measure  of  the  degree  to  which  a  system  is  aDle  to  satisfy  its 
performance  objectives. 

Cardinal  (interval)  Scale 

(ROWE) 

A  continuous  scale  between  two  end  points,  neither  of  wnicn  is 
necessari ly  fixed. 

Complexity  Level 

(CURRENT) 

The  general  degree  of  difficulty  to  complete  a  maintenance  request: 
high,  medium,  low. 

Computer  Program 

(AFR800- 14) 

A  series  of  instructions  or  statements  in  a  form  acceDtable  to  an 
electronic  comouter,  designed  to  cause  the  computer  to  execute  an 
operation  or  operations. 
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Computer  Resources 
(AFR800-14) 

The  totality  of  computer  equipment,  computer  programs,  associated 
documentation,  contractual  services,  personnel  and  supplies. 

Configuration  Control 

(AFR300-15) 

The  systematic  evaluation,  coordination,  approval  or  disapproval, 
and  implementation  of  approved  changes  in  the  configuration  of  a 
CPCI  after  formal  establishment  of  its  configuration  identification. 

Configuration  Item  (Cl) 

(AFR300-I5) 

An  item  of  ADPE  that  is  designated  for  configuration  management. 
(AFR800- 14) 

An  aggregation  of  equipment/software,  or  any  of  its  discrete  por¬ 
tions,  which  satisfies  an  end  use  function  and  is  designated  by  the 
Government  for  conf iguration  management.  CIs  may  vary  widely  in 
complexity,  size  and  type,  from  an  aircraft  or  electronic  system  to 
a  test  meter  or  round  of  ammunition.  During  development  and  initial 
production,  CIs  are  only  those  specification  items  that  are  refer¬ 
enced  directly  in  a  contract  (or  an  equivalent  in-house  agreement). 
During  the  CDeration  and  maintenance  period,  any  reparable  item 
designated  for  separate  procurement  is  a  configuration  item. 
(AFR  65-3) 

Configuration  Management  (CM) 

(AFR300-I5) 

A  management  discipline  that  applies  technical  and  administrative 
direction  and  surveillance  to: 

(1)  Identify  and  document  the  functional  and  pnysicai  charac¬ 
teristics  of  a  configuration  item. 

(2)  Control  changes  to  those  characteristics. 

(3)  Record  and  report  configuration  status. 

Configuration  Management  Plan  (CMP) 

(AFR300-I5) 

A  document  which  describes  project  responsibilities  and  procedures 
for  implementing  CM. 

Configuration  Management  System  (CMS) 

(AF0TECP5) 

A  system  applying  technical  and  administrative  direction  and  sur¬ 
veillance  to  identify  and  document  the  functional  and  physical 
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characteristics  of  a  configuration  item;  to  control  changes  to  those 
characteristics  and  to  record  and  report  change  processing  and 
implementation  status. 

Consequence  Value 

(ROWE) 

The  importance  a  risk  agent  subjectively  attaches  to  the  undesir¬ 
ability  of  a  specific  risk  consequence. 

Consensus 

(ROWE) 

Group  solidarity  in  sentiment  and  belief. . .general  agreement. 


Cost 


(ROWE) 

A  result  of  a  specific  action  that  constitutes  a  decrease  in  the 
production  possibilities  or  welfare  level  of  society.  Also  see 
Loss. 

Cost-Benefit  Analysis 
(ROWE) 

An  attempt  to  delineate  and  compare  in  terms  of  society  as  a  whole 
the  significant  effects,  both  positive  and  negative,  of  a  specific 
action.  Generally  a  number  of  alternative  actions  are  analyzed 
resulting  in  the  selection  of  the  alternative  that  provides  either 
the  largest  benefit-cost  ratio  (total  benef it/tota  1  cost)  or  one 
with  a  positive  ratio  at  least.  If  an  alternative  results  in  a  net 
benefit  less  than  zero  or  a  benefit-cost  ratio  less  than  1,  it  is 
deemed  socially  inefficient  and  is  not  carried  out. 

Cost-Effectiveness  Analysis 

(ROWE) 

A  term  less  specific  than  cost-benefit  analysis,  usually  meaning  the 
selection  of  the  lowest  cost  alternative  that  achieves  a  predeter¬ 
mined  level  of  benefits.  Alternatively,  the  analysis  and  selection 
of  the  path  that  yields  the  largest  social  benefit  for  a  predeter¬ 
mined  specified  level  of  social  costs. 

Critical  Automatic  Data  Processing  Resources 

(AFR205X) 

Those  resources  that  must  be  protected  because  their  compromise, 
alternation,  destruction,  loss,  or  failure  to  meet  objectives  will 
jeopardize  the  accomplishment  of  an  Air  Force,  Air  Force  subelement, 
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or  other  service  mission  or  the  accomplishment  of  DoO  life  support 
functions. 

Critical  Design  Review  (CDR) 

(AFR300-15) 

A  formal  review  conducted  during  the  development  phase  before  trans¬ 
lating  logic,  and  algorithms  to  coded  instructions. 

Critical  Issues 

(AF0TECP1 ) 

Those  aspects  of  a  system's  capability,  either  operational,  techni¬ 
cal,  or  other,  that  must  be  questioned  before  a  system's  overall 
worth  can  be  estimated  and  that  are  of  primary  importance  to  the 
decision  authority  in  reacning  a  decision  to  allow  the  system  to 
advance  into  the  next  acquisition  phase.  (DoO  Directive  5000.3). 

Data  Item  Description 

(AFR800-14) 

A  form  which  specifies  an  item  of  data  required  to  be  furnished  by  a 
contractor.  This  form  specifically  defines  the  content,  preparation 
instructions,  format  and  intended  use  of  each  data  product. 
(AFR  310-1) 

Decision  Analysis 

(ROWE) 

A  methodology  of  decomposition  of  the  decision-making  process  into 
parts,  wnereby  the  appropriate  data  can  be  associated  with  the 
parts,  to  provide  a  rational  basis  for  decision  making. 

Decision  Making 

(ROWE) 

A  dynamic  process  of  interaction,  involving  information  and  judgment 
among  participants  who  determine  a  particular  do! icy  choice. 
Decision  models  are  either  models  of  the  aecision-making  process 
itself,  or  analytical  models  (e.g.,  decision  trees,  decision  matri¬ 
ces)  used  as  aids  in  arriving  at  the  decisions.  Decision  theories 
usually  are  in  relation  to  the  process  itself. 

Decision  Matrices 

(ROWE) 

Matrices  wnose  elements  exhibit  Quantitative  relationships  (cardinal 
or  ordinal)  among  sets  of  factors  coming  intc  play  in  the  decision- 
making  process. 
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Decision  Tree 
(ROWE) 

A  device  used  to  portray  alternative  rses  of  action  and  relate 
them  to  alternative  decisions  showing  all  consequences  of  the 
decision.  The  tree  represents  alternative  courses  or  series  of 
actions  related  to  a  previous  decision. 

Decisive  Decision  Conditions 

(ROWE) 

Conditions  in  which  the  preference  between  values  on  a  utility  scale 
is  clearly  discernible  because  ranges  of  uncertainty  of  the  two 
values  do  not  overlap  (in  the  case  of  uniform  distributions  of 
uncertainty)  or  are  below  a  certain  error  level  (for  normal  distri¬ 
butions  of  uncertainty). 

Degree  of  Uncertainty 

(ROWE) 

That  proportion  of  information  about  a  total  system  that  is  unknown 
in  relation  to  the  total  information  about  the  system. 

Delphi  Technique 

(ROWE) 

An  iterative  method  designed  to  produce  a  consensus  by  repeated 
queries  of  an  individual  with  feedback  of  grouD  resoonses.  Members 
of  the  group  do  not  interact  directly. 

Descriptive  Uncertainty 

(ROWE) 

The  absence  of  information  about  the  comoleteness  of  the  description 
of  the  degrees  of  freedom  of  a  system. 

Designated  Approving  Authority 

(AFR205X) 

An  official  designated  to  approve  the  operation  of  automatic  data 
processing  systems  at  the  automatic  data  processing  facilities  unoer 
his  or  her  jurisdiction  for  storage  of  classified  or  sensitive 
unclassified  information  or  for  critical  processing. 

Deviation 

(AFR300- 15) 

A  written  authorization,  granted  prior  to  the  development  of  a  CPC  I , 
to  depart  from  a  particular  performance  or  design  retirement;  a 
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specification  for  a  specific  number  of  units;  a  specific  period  of 
time;  or  established  standards. 

Documentation 

(AF0TECP5) 

All  of  the  written  work  describing  operating  and  maintenance  proced¬ 
ures  for  a  system. 

Documentation  Consistency 

(AF0TECP5) 

A  measure  of  the  consistency  in  the  information  provided  in  support 
system  documentation. 

Documentation  Descriptiveness 

(AF0TECP5) 

A  measure  of  the  descriptiveness  of  the  information  provided  in 
support  system  documentation. 

Documentation  Modularity 

(AF0TECP5) 

A  measure  of  the  modular  organization  of  information  provided  in 
support  system  documentation. 

Documentation  Simplicity 

(AF0TECP5) 

A  measure  of  the  ease  of  use  and  lack  of  complexity  in  the  informa¬ 
tion  provided  in  computer  system  documentation. 

EmDedded  Computer  Resources 

(AF0TECP1) 

Computer  resources  incorDorated  as  integral  parts  of,  dedicated  to, 
required  for  direct  support  of,  or  for  the  uogradir.g  or  modification 
of  major  or  less  than  major  system(s).  (Excludes  AOP  resources  as 
defined  and  administered  under  AFR  300  series.)  (USAF/RD/LE  Policy 
letter,  13  October  198 L ) . 

Embedded  Computer  System  (ECS) 

(AF0TECP1) 

a)  A  computer  chat  is  Integra!  to  an  electromechanical  system  ana 
that  has  the  following  <ey  attributes: 

(1)  Physically  incorporated  into  a  large  system  wnose  primary 
function  is  not  aata  processing. 
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(2)  Integral  to,  or  supportive  of,  a  larger  system  from  a 
design,  procurement,  and  operations  viewpoint. 

(3)  Inputs  include  target  data,  environmental  data,  command 
and  control,  etc. 

(4)  Outputs  include  target  information,  flight  information, 
control  signals,  etc. 

b)  In  general,  an  embed  led  computer  system  (ECS)  is  developed, 
acquired,  and  operated  under  decentral ized  management.  (OoD  Direc¬ 
tives  5000.1,  5000.2). 

(AF0TECP5) 

A  computer  that  is  integral  to  an  electronic  or  electromechanical 
system  (e.g.,  aircraft,  missile,  spacecraft,  communications  device) 
from  a  design,  procurement,  and  operational  viewpoint. 

Empirical 

(ROWE) 

Originating  in  or  based  on  observation  or  experience. 

Equitable  Risk 
(ROWE) 

A  risk  agent  receives  direct  benefits  as  a  result  of  exposure  to  a 
risk,  and  the  knowledge  of  the  risk  is  not  purposely  withheld  from 
the  risk  agent. 

Estimation 

(ROWE) 

The  assignment  of  probability  measures  to  a  postulated  future  event. 
Estimator  Uncertainty 
(ROWE) 

Uncertainty  in  measurement  resulting  from  deliberate  use  of  less 
complex  measures  such  as  central  value  estimates  of  dispersion  and 
smoothing  functions  for  time-dependent  parameters. 

Evaluation 

(ROWE) 

Comparison  of  performance  of  an  activity  with  the  objectives  of  the 
activity  ana  assignment  of  a  success  measure  to  that  performance. 

Evaluation  Criteria 

(AF0TECP1) 

Standards  by  wnich  achievement  of  requireu  operational  effective¬ 
ness/suitability  character i sties  or  resolution  of  technical  or 
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operational  issues  may  be  judged.  For  full-scale  development  and 
beyond,  evaluation  criteria  must  include  quantitative  goals  (the 
desired  value)  and  thresholds  (the  value  beyond  which  the  character¬ 
istic  is  unsatisfactory)  whenever  possible.  (DoO  directive  5000.3). 


Event 


(ROWE) 

A  particular  point  in  time  associated  with  the  beginning  or  comple¬ 
tion  of  an  activity,  and  possibly  accompanied  by  a  statement  of  the 
benefit  or  result  attained  or  to  be  attained  because  of  the  comple¬ 
tion  of  an  activity. 

Expandabi 1 ity 

(AF0TECP5) 

A  measure  of  the  ease  with  which  the  functional  capability  of  com¬ 
puter  hardware  or  software  may  be  expanded. 

Expected  Value,  Use  Of 

(ROWE) 

Valuation  of  an  uncertain  numerical  event  by  weighting  all  possible 
events  by  their  probability  of  occurrence  and  averaging. 

Expert  Judgment 

(ROWE) 

Designating  the  relevance  of  opinions  of  persons  well  informed  in  an 
area  for  estimates  (e.g.,  forecasts  of  economic  activity). 

Exposure  (to  risk) 

(ROWE) 

The  condition  of  being  vulneraple  to  seme  degree  to  a  particular 
outcome  of  an  activity,  if  that  outcome  occurs. 

Extrapolation/Pro ject  ion 

(ROWE) 

The  technique  of  estimating  the  future  oy  a  continuation  of  past 
trends  without  attempts  to  understand  the  underlying  phenomena. 

Faci 1 i ty 

(AF0TECP5) 

The  physical  plant  and  the  services  it  provides;  specific  examples 
are  pnysical  space,  electrical  power,  physical  and  electromagnetic 
(TEMPEST)  security,  environmental  control,  fire  safety  provisions, 
and  communications  availability. 
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Feasible 

(ROWE) 

That  which  is  possible  to  do,  realistically. 

Feedback 

(ROWE) 

The  return  of  performance  data  to  a  point  permitting  comparison  with 
objective  data,  normally  for  the  purpose  of  improving  performance 
(goal-seeking  feedback),  but  occasionally  to  modify  the  objective 
(goal-changing  feedback). 

F i rmware 

(AF0TECP1) 

a)  Computer  programs  and  data  loaded  in  a  class  of  memory  that 
cannot  be  dynamically  modified  by  the  computer  during  processing. 

b)  Hardware  that  contains  a  computer  program  and  data  that  cannot  be 
changed  in  its  application  envi-onment. 

Note  1.  The  compute*-  programs  and  data  contained  in  firmware  are 
classified  as  software;  the  circuitry  containing  the  computer  pro¬ 
gram  and  data  is  classified  as  hardware.  (Data  and  Analysis  Center 
for  Software) . 

Functional  Configuration  Audit  (FCA) 

(AFR300- 15) 

The  formal  examination  of  CPCI  to  verify  that  tne  performance  speci¬ 
fied  in  the  SS  has  oeen  achieved. 

Independent  Verification  and  Validation  (IV&V) 

(AF0TECP1) 

An  independent  assessment  process  structured  to  ensure  that  compute*' 
programs  fulfill  the  requirements  stated  in  system  ana  suosystem 
specifications  and  satisfactori ly  perform  the  functions  required  to 
meet  the  user's  and  supoorter's  requirements.  IV&V  consists  of 
three  essential  elements:  independence,  verif ication,  and  valida¬ 
tion: 

(1)  Independent.  An  organization/agency  wnich  is  separate 
from  the  software  development  activity  from  a  contractual 
and  organi zat iona i  standpoint. 

(2)  Verification.  ~ne  evaluation  to  determine  whether  the 
products  of  eacn  step  of  the  computer  Digram  development 
process  fulfill  ail  requirements  levied  by  the  previous 
step. 

(3)  Validation.  'he  integration,  testing,  and/or  evaluation 
activities  carried  out  at  the  system/subsystem  level  to 
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evaluate  the  developed  computer  program  against  the  system 
specifications  and  the  user's  and  supporter's  require¬ 
ments.  (APR  88-14) 

Individual  Risk  Evaluation 

(ROWE) 

The  complex  process,  conscious  or  unconscious,  whereby  an  individual 
accepts  a  given  risk. 

Inequitable  Risk 

(ROWE) 

A  risk  agent  is  exposed  to  a  risk  and  receives  no  direct  benefits 
from  such  exposure,  or  the  knowledge  of  the  risk  is  purposely  with¬ 
held  from  him. 

Interdependence 

(ROWE) 

A  Droperty  shared  by  two  or  more  entities  whenever  the  performance 
of  any  one  affects  the  performance  of  some  or  all  the  rest. 

Interoperabi  1  i  ty 

(AF0TECP5) 

A  measure  of  the  degree  to  wnicn  computer  Hardware  or  software  can 
interface  to  and  operate  with  other  similar  computer  hardware  or 
sof tware. 


Intrinsic  Parameter 


(ROWE) 

A  variable  whose  measurement  is  based  on  the  value  system  of  an 
individual  and  his  perception  of  these  values. 

Loss  Function 


(ROWE) 

A  function  used  in  decision  theory  for  evaluating  the  losses  incur¬ 
red  when  certain  decisions  are  made  under  uncertainty.  If  the  loss 
function  is  independent  of  the  decision  value  used,  it  is  frequently 
called  a  cost  function. 


Maintainability 
(AF0TECP3 ) 

Those  cnaracteristics  of  software  which  affect  the  anility  of  the 
software  programmer  to  correct  errors,  enhance  system  capaoilities 
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through  software  changes,  and  modify  the  software  to  be  compatible 
with  hardware  changes. 

(AF0TECP5) 

The  probability  that  a  system  out  of  service  for  maintenance  can  be 
properly  repaired  and  returned  to  service  in  a  stated  elapsed  time. 

Maintenance  Documentation 

(AF0TECP5) 

The  documentation  that  describes  the  maintenance  of  computer  system 
hardware  and  software. 

Maintenance  Request  Category 

(CURRENT) 

The  identification  of  a  maintfinance  request  by  specification  of  the 
priority  type,  maintenance  type,  and  complexity  level. 

Maintenance  Type 

(CURRENT) 

The  type  of  maintenance  actions  required  to  complete  a  maintenance 
request:  ennancement,  conversion,  correction. 

Measurable 

(ROWE) 

a)  Capable  of  being  sensed,  that  which  is  sensed  being  convertible 
to  an  indication;  the  indication  can  be  logical,  axiological,  numer¬ 
ical,  or  probaoi 1 istic.  If  probabilistic,  it  is  empirical  and  sub¬ 
jective. 

b)  Comparable  to  some  unit  designated  as  standard. 

Measured  Risk  Level 

(ROWE) 

The  historic,  measured,  or  modeled  risk  associated  with  a  given 
activity. 

Measurement  Uncertainty 
(ROWE) 

The  absence  of  information  aoout  the  specific  value  of  a  measurable 
variaple. 

Methodology 

(ROWE) 

An  open  system  of  procedures. 
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Model 

(ROWE) 

An  abstraction  of  reality  that  is  always  an  approximation  to 
real ity. 

Module 

(AFR300-15) 

A  program  unit  that  is  discrete  and  identifiable  with  respect  to 
compiling  and  combining  with  other  units. 

Nominal  Scale  (taxonomy) 

(ROWE) 

A  classification  of  items  that  can  be  distinguished  from  one  another 
by  one  or  more  properties. 

Objective  Function 

(ROWE) 

A  specified  mathematical  relationship  between  a  dependent  variable 
(e.g.,  overall  measure  of  benefits)  and  a  set  of  independent  vari¬ 
ables  (e.g.,  individual  benefit  measures  and  their  relative 
weights).  In  choosing  among  alternatives,  the  decision  maker 
typically  seeks  to  maximize  tne  (dependent  variable  of  the)  objec¬ 
tive  function. 

Operational  Effectiveness 

(AF0TECP1) 

The  overall  degree  of  mission  accomplishment  of  a  system  used  by 
representative  personnel  in  the  context  of  the  organization, 
doctrine,  tactics,  threat  (including  countermeasures  and  nuclear 
threats),  and  environment  in  the  plained  operational  employment  of 
the  system.  (OoO  Directive  5000.3) 

Operational  Suitability 

(AF0TECP1) 

The  degree  to  which  a  system  can  be  sati  sf  actori  ly  placed  in  field 
use,  with  consideration  being  given  availability,  compatibility, 
transportability,  interoperabi  1  i  ty ,  reliability,  wartime  usage 
rates,  maintainability,  safety,  human  factors,  manpower  supportabil- 
ity,  logistic  supportab i 1 i ty ,  and  training  requirements.  (OoO 
Directive  5000.3) 
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Opinion  Survey/Sampl ing 
(ROWE) 

Any  procedure  for  obtaining  by  oral  or  written  interrogation  or  both 
the  views  of  any  portion  of  the  affected  population  regarding 
benefit  levels  expected,  their  utility,  and/or  relative  importance. 
Typically,  scientific  sampling  procedures  would  be  used  to  maximize 
(for  a  given'  level  of  effort)  the  accuracy  and  precision  of  the 
results  obtained. 

Opportunity  Cost 

(ROWE) 

The  value  to  society  of  the  next  best  alternative  use  of  a  resource. 
This  is  the  true  economic  cost  to  society  of  using  a  resource  for  a 
specific  purpose  or  in  a  specific  project. 

Ordinal  Scale  (rank  scale) 

(ROWE) 

An  ordering  (ranking)  of  items  by  the  degree  to  which  they  satisfy 
some  criterion. 

Paradigm 

(ROWE) 

A  structured  set  of  concepts,  definitions,  classifications,  axioms, 
and  assumptions  used  in  providing  a  conceptual  framework  for  study¬ 
ing  a  given  proolem. 

Parametric  Variation 

(ROWE) 

A  technique  for  sensitivity  analysis  of  any  given  model  in  which  the 
values  of  parameters  that  are  input  to  the  model's  calculation  are 
systematically  varied  to  permit  observation  of  how  such  variation 
affects  the  model's  output  (especially  ranking  of  alternatives). 

Personnel 

(AF0TECP5) 

A  general  term  for  the  experience,  education,  and  quantity  of  people 
who  are  assigned  to  the  software  support  facility  either  directly  or 
indirectly  maintaining  the  ECS.  It  includes  Management,  Technical, 
Support,  and  Contractor  resources. 

Personnel  Profile 

(AF0TECP5) 

The  characteristics  that  describe  the  experience,  education,  and 
quantity  of  software  support  facility  personnel. 
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Physical  Configuration  Audit  (PCA) 

( AFR300-15 ) 

the  formal  examination  of  the  coded  version  of  a  computer  program 
configuration  item  against  its  technical  documentation. 

Precision 

(ROWE) 

The  exactness  with  which  a  quantity  is  stated;  that  is,  the  number 
of  units  into  which  a  measurement  scale  of  that  quantity  may  be 
meaningfully  divided.  The  number  of  significant  digits  is  a  measure 
of  precision. 

Predictive  Modeling 

(ROWE) 

Use  of  any  mathematic  model  that  estimates  o-  predicts  the  value  of 
a  dependent  variable  in  terms  of  component  factors  specified  as 
independent  variables. 

Preference 

(ROWE) 

Assignment  of  rank  to  items  by  an  agent  when  the  criterion  used  is 
utility  to  the  ranking  agent. 

Priority  Type 

(CURRENT) 

The  criticality  of  the  maintenence  request  in  order  to  perserve 
mission  readiness:  emergency,  urgent,  normal. 

Probab  i  1  i ty 

(ROWE) 

A  numerical  property  attached  to  an  activity  or  event  whereby  the 
likelihood  of  its  future  occurrence  is  expressed  or  clarified. 

Probability  Distribution 

(ROWE) 

The  representation  of  a  repeatable  stochastic  process  by  a  function 
satisfying  the  axioms  of  probability  theory.' 

Probability  of  Occurrence 

(ROWE) 

The  probability  that  a  particular  event  will  occur,  or  will  occur  in 
a  given  interval. 
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Probability  Threshold 
(ROWE) 

A  probability  of  occurrence  level  for  a  risk  below  which  a  risk 
agent  is  no  longer  concerned  with  the  risk  and  ignores  it  in  prac¬ 
tice  (Threshold  of  concern). 

Product  Baseline 

(AFR300-15) 

The  initial  approved  product  configuration  identification. 

Product  Verification  Review  (PVR) 

( AFR300- 15) 

A  formal  review  conducted  by  the  developer  for  each  CPCI  at  the  end 
of  the  development  Dhase  to  establish  the  Product  Baseline  for  that 
CPCI  and  to  ensure  preparation  for  tne  Test  Phase  has  been  com¬ 
pleted. 

Program  Manager 
(AFR800-I4) 

The  generic  term  used  to  denote  a  single  Air  Force  manager  (System 
Program  Director,  Program/Pro ject  Manager,  or  System/Item  Manager) 
during  any  specific  phase  of  the  acquisition  life  cycle. 
(AFR  800-2). 

Program  Management  Directive  (PMD) 

(AFR800- 14) 

The  official  HQ  USAF  management  directive  used  to  provide  direction 
to  the  implementing  and  participating  commands  and  satisfy  documen¬ 
tation  reauirements.  It  will  be  used  during  the  entire  acquisition 
cycle  to  state  requirements  and  request  studies  as  well  as  initiate, 
approve,  change,  transition,  modify  or  terminate  programs.  The  con¬ 
tent  of  the  PMD,  including  the  required  HQ  USAF  review  and  approval 
actions,  is  tailored  to  the  needs  of  each  individual  program. 
(AFR  800-2) 

Program  Management  Plan  (PMP) 

(AFR800-14) 

The  document  developed  and  issued  by  the  Program  Manager  which  shews 
the  integrated  time-phased  tasxs  and  resources'  required  to  complete 
the  task  soecified  in  the  2M0.  The  PMP  is  tailored  to  the  needs  of 
each  individual  program.  (AFR  300-2) 
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Program  Office  (PO) 

(AFR800-14) 

The  field  office  organized  by  the  Program  Manager  to  assist  him  in 
accomplishing  the  program  tasks.  (AFR  800-2) 

Program  Support  Tools 

(AF0TECP3) 

General  debug  aids,  test/retest  software,  trace  sof tware/hardware 
features,  use  of  compi ler/1 ink  editor,  library  management/configura¬ 
tion  management/text  editor/display  software  tools. 

Program  Test  Plan 

(AF0TECP3) 

Set  of  descriptions  and  procedures  for  hew  the  program  is  to  be  (or 
can  be,  or  has  been)  tested. 

ProDensity  for  Risk  Acceptance 

(ROWE) 

An  individual,  subjective  trait  designating  the  degree  of  risk  one 
is  willing  to  subject  himself  to  for  a  particular  purpose. 

Quality  Assurance  (QA) 

(AFR  300- L  5 ) 

All  actions  that  are  taken  to  assure  that  a  development  organization 
delivers  products  that  meet  performance  -‘equirements  and  adhere  to 
standards  and  procedures. 

Quantification 

(ROWE) 

The  assignment  of  a  number  to  an  entity  or  a  method  for  determining 
a  number  to  be  assigned  to  an  entity 

Re  1 i ab i 1 i ty 

(ROWE) 

The  probability  that  the  system  will  perform  its  reouired  functions 
under  given  conditions  for  a  specified  operating  time. 

Residual  Risk 

(AFR205X) 

That  portion  of  risk  which  remains  after  security  measures  have  been 
appl  ied. 
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(AFR205X) 

The  loss  potential  which  exists  as  the  result  of  threat/vulnerabil¬ 
ity  pairs.  Reducing  either  the  threat  or  the  vulnerability  reduces 
the  risk. 

(ROWE) 

The  potential  for  realization  of  unwanted,  negative  consequences  of 
an  event. 

Risk  Acceptance 

(ROWE) 

Willingness  of  an  individual,  group,  or  society  to  accept  a  soecific 
level  of  risk  to  obtain  seme  gain  or  benefit. 

Risk  Acceptance  Function 

(ROWE) 

A  subjective  operator  relating  the  levels  of  probability  of  occur¬ 
rence  and  value  of  a  consequence  to  a  level  of  risk  acceptance. 

Risk  Acceptance  Level 

(ROWE) 

The  acceptaDle  probability  of  occurrence  of  a  specific  consequence 
value  to  a  given  risk  agent. 

Risk  Acceptance  Utility  Function 

(ROWE) 

The  profile  of  the  acceptability  of  the  probability  of  occurrence 
for  ail  consequences  involved  in  a  risk  situation  for  a  specific 
risk  agent. 

Risk  Agent 

(ROWE) 

See  Valuing  Agent. 

Risk  Analysis 
(AFR205X) 

A  part  of  risk  management:  that  is  used  to  minimize  risk  by  effec¬ 
tively  applying  security  measures  commensurate  with  the  relative 
threats,  vulnerabilities,  and  values  of  the  resources  to  be 
orotected.  (The  value  of  the  resources  includes  impact  ci  the 
organizations  the  automatic  data  processing  system  supports,  and 
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impact  of  the  loss  or  unauthorized  modification  of  data).  Risk 
analysis  may  be  thought  of  as  consisting  of  four  modules:  sensitiv¬ 
ity  assessment,  risk  assessment,  economic  assessment,  and  security 
test  and  evaluation. 

Risk  Assessment 

(AFR205X) 

A  detailed  study  of  the  vulnerabilities,  threats,  likelihood,  loss 
or  impact,  and  theoretical  effectiveness  of  security  measures.  The 
results  of  a  risk  assessment  may  be  used  to  develop  security 
requirements  and  specifications. 

(ROWE) 

The  total  process  of  quantifying  a  risk  and  finding  an  acceptable 
level  of  that  risk  for  an  individual,  group,  or  society.  It 
involves  both  risk  determination  and  risk  evaluation. 

Risk  Averse 

(ROWE) 

Displaying  a  propensity  against  taking  risks. 

Risk  Aversion 
(ROWE) 

The  act  of  reducing  risk. 

Risk  Baseline 
(CURRENT) 

The  risk  probability  density  function  and  the  associated  magnitude 
of  consequence  for  the  potential  negative  outcomes. 

Risk  Consequence 

(ROWE) 

The  impact  to  a  risk  agent  of  exposure  to  a  risky  event. 

Risk  Conversion  Factor 
(ROWE) 

A  numerical  weight  allowing  one  type  of  risk  to  be  compared  to 
another  type. 

Risk  Determination 

(ROWE) 

The  process  of  identifying  ana  estimating  the  magnitude  of  risk. 
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Risk  Estimation 
(ROWE) 

The  process  of  quantification  of  the  probaDi 1 i ties  and  consequence 
values  for  an  identified  risk. 

Risk  Evaluation 

(ROWE) 

The  complex  process  of  developing  acceptable  levels  of  risk  to 
individuals  or  society. 

Risk  Evaluator 

(ROWE) 

A  person,  group,  or  institution  that  seeks  to  interpret  a  valuing 
agent's  risk  for  a  particular  purpose. 

Risk  Identification 

(ROWE) 

The  observation  and  recognition  of  new  risk  parameters,  or  new 
relationships  among  existing  risk  parameters,  or  perceotion  of  a 
change  in  the  magnitude  of  existing  risk  parameters. 

Risk  Management 

(AFR205X) 

The  total  process  of  identifying,  controlling,  and  minimizing 
uncertain  events.  The  process  of  obtaining  and  maintaining  OAA 
approval  is  a  major  element  of  the  risk  management  program.  The 
process  facilitates  the  management  of  automatic  data  processing 
(ADP)  security  risks  by  each  level  of  AOP  management  throughout  the 
ADP  life  cycle.  The  approval  process  consists  of  three  elements: 
risk  analysis,  certification,  and  approval. 

Risk  Profile  Basel ine 

(CURRENT) 

The  measure  of  information  and/or  requirements  which  serve  as  the 
zero  reference  against  which  negative  (and  positive)  outcomes  can  oe 
determined. 

Risk  Proportionality  Derating  Factor 
(ROWE) 

Quantifying  the  degree  to  which  risks  become  less  acceptable  as 
indirect  benefits  to  the  risk  agent  declines. 
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Risk  Proportionality  Factor 
(ROWE) 

that  portion  of  the  total  societal  risk  that  society  will  accept  for 
a  new  technology. 

Risk  Reduction 

(ROWE) 

The  action  of  lowering  the  probability  of  occurrence  and/or  the 

value  of  a  risk  consequence,  thereby  reducing  the  magnitude  of  the 
risk . 

Risk  Reference 
(ROWE) 

Some  reference,  absolute  or  relative,  against  which  the  acceptabil¬ 
ity  of  a  similar  risk  may  be  measured  or  related;  implies  some 

overall  value  of  risk  to  society. 

Risk  Referent 

(ROWE) 

A  specific  level  of  risk  deemed  acceptable  by  society  or  a  risk 

evaluator  for  a  specific  risk;  it  is  derived  from  a  risk  reference. 

Risky  Shift 

(ROWE) 

The  tendency  of  certain  groups  to  become  more  extreme  or  take 
riskier  positions  in  their  judgments  than  they  would  acting  as 

individuals. 

Sensitivity  Analysis 

(ROWE) 

A  method  used  to  examine  the  operation  of  a  system  by  measuring  the 
deviation  of  its  nominal  behavior  due  to  perturbat ions  in  the  per¬ 
formance  of  its  components  from  their  nominal  values. 

Simulation 

(AFR800-14) 

The  representation  of  physical  systems  or  phenomena  by  computers, 
models  or  other  equipment. 

Software 

(AF0TECP1) 

A  set  of  computer  programs,  procedures,  and  associated  documentation 
concerned  with  the  operation  of  a  data  processing  system. 
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(CURRENT) 

The  programs  which  execute  in  a  computer.  The  data  input,  output, 
and  controls  upon  which  program  execution  depends  and  the 
documentation  which  describes,  in  a  textual  medium,  development  and 
maintenance  of  the  programs. 

Software  Error 

(CURRENT) 

The  human  decision  (inadvertent  or  by  design)  which  results  in  the 
inclusion  of  a  fault  in  a  software  product. 

Software  Fault 

(CURRENT) 

The  presence  or  absence  of  that  part  of  a  software  product  which  can 
result  in  software  failure. 

Software  Maintainability 

(AF0TECP1 ) 

The  ease  with  which  software  can  be  changed  in  order  to: 

(1)  Correct  errors. 

(2)  Add  or  modify  system  capabilities  through  software 
changes. 

(3)  Delete  features  from  programs. 

(4)  Modify  software  to  be  compatible  with  hardware  changes. 
(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  perform 
software  maintenance  actions. 

Software  Maintenance 

(CURRENT) 

Those  actions  required  for: 

(1)  Correction.  Removal,  correction  of  software  faults 

(2)  Enhancement.  Addition/deletion  of  features  from  the 
software 

(3)  Conversion.  Modification  of  the  software  because  of 
environment  (data  hardware)  changes. 

Software  Maintenance  Environment 

(CURRENT) 

An  integration  of  personnel  support  systems  and  physical  facilities 
for  the  purpose  of  maintaining  software  products. 
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Software  Maintenance  Measures 
(CURRENT) 

Measures  of  software  maintainability  and  environment  capabilities  to 
support  software  maintenance  activity. 

Software  Management 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  development/maintenance  activi¬ 
ties.  Also,  those  personnel  with  software  management  responsi- 
I  b  i  1  i  t  i  e  s . 

Software  Reliabi lity 

(CURRENT) 

A  quality  of  software  which  reflects  the  probability  of  failure  free 
operation  of  a  software  component  or  system  in  a  specified  environ¬ 
ment  for  a  specified  time. 

Software  Portability 

(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  transfer 
the  software  from  one  environment  (hardware  and  system  software)  to 
another. 

Software  Support  Facility  (SSF) 

(AF0TECP5) 

The  facility  which  houses  and  provides  services  for  the  support 
systems  and  personnel  required  to  maintain  the  software  for  a 
specific  ECS. 

Software  Supportabi  1  ity 

(CURRENT) 

A  measure  of  the  adequacy  of  personnel,  resources,  and  procedures  to 
faci  1 itate: 

(1)  Modifying  and  installing  software 

(2)  Establishing  an  operational  software  baseline 

(3)  Meeting  user  requirements. 

Software  Supportab i 1 ity  Evaluation  Metrics 
(CURRENT) 

The  closed-form  questionnaire  scores  for  each  characteristic  and 
cumulated  level  in  a  software  supportabi 1 ity  evaluation. 
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Software  Supportabi 1 i ty  Magnitude  of  Risk  Consequence 
(CURRENT) 

The  level  of  impact  to  a  software  user  or  supporter  as  a  result  of 
the  risk  level  of  a  software  supportabi 1 ity  negative  outcome. 

Software  Supportabi 1 ity  Negative  Outcome. 

(CURRENT) 

The  final  result  of  a  maintenance  request  as  represented  by  the  pair 
(time  to  comp’ete  request,  number  of  requests  per  unit  time),  in 
which  the  Baseline  SS  Profile  is  not  met. 

Software  Supportabi 1 i ty  Risk  Agent  Acceptance  Level 

(CURRENT) 

The  software  supportabi 1 i ty  risk  level  which  is  acceptable  to  a  risk 
agent. 

Software  SupDortability  Risk  Level. 

(CURRENT) 

The  potential  for  realization  of  a  software  supportaoi i i ty  negative 
outcome. 

Specification 

(AFR300-15) 

A  document  that  describes  the  requirements  for  the  development  or 
acquisition  of  ADPE  and/or  software. 

Standards 

(AF0TECP3) 

Procedures,  rules,  and  conventions  used  for  prescribing  disciplined 
program  design  and  implementation. 

States  of  Nature 

(ROWE) 

A  concept  from  decision  theory.  [n  decision  making  under  uncer¬ 
tainty,  the  outcomes  (numerical  results)  associated  witn  eacn  avail¬ 
able  alternative  are  considered  to  be  predictable  as  a  set  of  n  dis¬ 
crete  values  depending  on  conditions  beyond  the  decision  maker's 
control  and  for  which  "e  nas  no  useful  estimates  of  the  respective 
probabilities.  The  n  sets  of  conditions  under  wnich  each  one  of  the 
outcomes  is  expected  are  termed  "states  of  nature." 
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Structured  Value  (structured  value  analysis) 

(ROWE) 

The  resultant  value  of  a  particular  value  set  evaluated  for  a  par¬ 
ticular  data  set.  This  value  lies  between  zero  and  unity  and  allows 
many  data  sets  to  be  ranked  numerically  in  relation  to  one  another. 

Structured  Value  Analysis 
(ROWE) 

A  multistage  procedure  for  assessing  the  value  of  an  action,  project 
alternative,  and  so  on,  incorporating  individual  techniques  at  each 
stage  for  computing  from  quantitative  measures  of  individual  com¬ 
ponents  a  single  figure  expressing  the  overall  value.  A  multistage 
procedure  for  assessing  the  value  of  an  action,  project,  alterna¬ 
tive,  and  so  on,  by  structuring  the  complete  entity  into  component 
elements,  to  each  of  which  a  numeric  measure  of  value  (positive  or 
negative)  can  be  assigned.  These  are  then  converted  to  a  common 
utility  scale.  Each  component  is  assigned  a  weight  expressing  its 
relative  significance  in  determining  overall  value  of  the  entity.  A 
single  figure  of  worth  or  value  is  then  computed  from  measures  and 
weights  of  all  individual  components.  The  procedure  permits  con¬ 
siderable  flexibility  in  choice  of  techniques  used  to  perform  each 
necessary  optimal  step. 

Subjective  Probabilities 
(ROWE) 

The  assignment  of  subjective  weights  to  possible  outcomes  of  an 
uncertain  event  where  weights  assigned  satisfy  axioms  of  probability 
theory. 

upport  Personnel 
( AF0TECP5 ) 

A  general  term  for  military  or  DoD  civilian  personnel  whose  skills 
are  necessary  for  the  software  support  facility  to  function  but  who 
do  not  directly  support  ECS  software  maintenance. 

Support  System 
(AF0TECP5) 

Any  automated  system  used  to  change,  test,  or  manage  the  configur¬ 
ation  of  ECS  software  and  associated  documentation.  Includes  but  is 
not  limited  to  Host  Processor,  Software  3ench,  Laboratory-Integrated 
Test  Facility,  Operationa 1  -  Integrated  Test  Facility,  and  Configura¬ 
tion  Management  System. 
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Support  System  Facility 
(AF0TECP5) 

The  facility  resources  that  must  be  available  for  the  software 
support  resources  to  accomplish  a  specific  task(s)  (see  General 
Faci 1 ity) . 

Surrogate  or  Proxy  Measures 
(ROWE) 

The  use  of  a  related  quantity  as  a  proxy  for  an  unknown  or  diffi- 
cult-to-measure  value.  The  relationship  may  be  established  by  arm¬ 
chair  analysis,  correlation  techniques,  scientific  studies,  or  other 
means. 

System 

(ROWE) 

a)  A  complex  entity  formed  of  many,  often  diverse,  parts  subject  to 
a  common  plan  or  serving  a  common  purpose. 

b)  A  composite  of  equipment,  skills,  and  techniques  capable  of  per¬ 
forming  and/or  supporting  an  operation. 

System  Design  Review  (SDR) 

(AFR300- 15) 

A  formal  review  of  the  system  design  approach  for  an  ADS. 

System  Requirements  Review  (SRR) 

(AFR300- 15) 

A  formal  review  cf  the  requ i rements  for  an  AOS. 

System  Software 
(AF0TECP5) 

All  of  the  software  that  is  part  of  the  software  support  facility 
computer  system*  It  is  never  or  seldom  accessed  directly  by  soft¬ 
ware  support  facility  oersonnel;  it  controls  the  orocessing  of 
application  software.  It  includes  the  Operating  System,  Source  Code 
Editor,  Language  Translator,  Link  Editor/Loader,  Liorarian/Fi le 
Manager,  Data  Base  Manager,  and  Automated  Software  Development  Tool. 

Taxonomy 

(ROWE) 

The  identification  and  definition  of  properties  of  elements  of  the 
universe;  a  disaggregation,  as  contrasted  with  systematics  (wnich  is 
an  aggregation)  and  as  contrasted  with  morpnology  (which  encompasses 
both  taxonomy  and  systematics). 
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Test  Analysis  Report  (RT) 

(AFR300-15) 

A  document  containing  the  results  and  analyses  of  tests  executed 
during  the  Test  Phase. 

Threshold 

(ROWE) 

A  discontinuous  change  of  state  of  a  parameter  as  its  measure 
increases.  One  condition  exists  below  the  discontinuity,  and  a 
different  one  above  it. 

Time  to  Complete  Maintenance  Request  (TC) 

(CURRENT) 

The  calendar  time  from  receipt  of  the  maintenance  request  by  the 
support  control  group  until  the  request  has  been  denied  or  the 
maintenance  actions  required  by  request  have  been  accepted  as  part 
of  an  operational  system  software  configured  release.  (This  does 
not  mean  the  configuration  is  released  or  distributed,  and  this  time 
does  not  include  this  additional  delay  if  any.) 

Transfer 

( AFR800-14 ) 

That  point  in  time  when  the  designated  Supporting  Command  accepts 
program  management  responsibilities  from  the  Implementing  Command. 
This  includes  logistic  support  and  related  engineering  and  procure¬ 
ment  responsibilities.  (AFR  800-4) 

Turnover 

(AFR800-14 ) 

That  point  in  time  when  the  operating  command  formally  accepts 
responsibility  from  the  Implementing  Command  for  the  operation  and 
maintenance  of  the  system,  equipment,  or  computer  program  acquired. 
(AFR  800-19) 

Uncerta inty 

(ROWE) 

The  absence  of  information;  that  which  is  unknown. 

User 

(AFR205X) 

Any  persons  (or  organizations)  having  access  to  an  automatic  data 
processing  system  via  communication  through  a  remote  device  or  who 
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is  allowed  to  submit  input  to  the  system  through  other  media  (for 
example,  tape  or  card  decks).  (Does  not  include  those  persons  or 
organizations  defined  as  customers.) 

Valuation 

(ROWE) 

The  act  of  mapping  an  ordinal  scale  onto  an  interval  scale  ,i.e., 
assigning  a  numerical  measure  to  each  ranked  item  based  on  its 
relative  distance  from  the  end  points  of  the  interval  scale... 
assigning  an  interval  scale  value  to  a  risk  consequence. 

Value 


(ROWE) 

A  quality  quantified  on  a  scale  expressing  the  satisfaction  of  man's 
intrinsic  wants  and  desires. 

Value  Function  (structured  value  analysis) 

(ROWE) 

A  function  relating  points  on  the  parameter  measurement  scale  to  the 
value  scale  for  a  particular  parameter.  These  functions  may  result 
from  explict  information  or  may  be  arrived  at  through  value  judg¬ 
ment. 

Value  Set  (structured  value  analysis) 

(ROWE) 

A  specific  set  of  model  parameters  made  up  of  terms  and  factors, 
expressed  in  particular  measurement  scales,  value  functions,  and 
weights . 

Valuing 

(ROWE) 

The  act  of  assigning  a  value  to  a  risk  consequence. 

Valuing  Agent 
(ROWE) 

A  person  or  group  of  oersons  who  evaluates  directly  the  consequence 
of  a  risk  to  which  he  is  suDjectea.  A  risk  agent. 

yeri f ication/Val idation  (of  computer  orograms) 

(AFR800-14) 

The  process  of  determining  that  the  computer  program  was  develooed 
in  accordance  with  the  stated  specification  and  sat i sfactori ly  per¬ 
forms,  in  the  mission  environment,  the  function(s)  for.  which  it  was 
designed. 
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Weight  (structured  value  analysis) 

(ROWE) 

The  relative  importance  of  terms  in.  a  model  expressed  as  a  decimal 
fraction;  weights  for  a  set  of  terms  add  to  unity. 

Weighting  Factor 

(ROWE) 

A  coefficient  used  to  adjust  variable  accuracy  to  a  subjective 
evaluation;  these  factors  are  usually  determined  through  surveys, 
Delphi  sessions,  or  other  formats  of  expressing  social  priorities. 
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